REPORT OF THE MEVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) Submitted by: LTG Associates, Inc. and TvT Associates a division of Social & Scientific Systems, Inc. Submitted to: The United States Agency for International Development Under Contract No. HRN-I-00-99-00002-00, Technical Directive 49 March 2002 REPORT OF THE MID-TERM EVALUATION OF THE CHANGE PROJECT Moncef M. Bouhafa Neal Brandes Patricia Hammer Iain McLellan Submitted by: LTG Associates, Inc. and TvT Associates, a division of Social & Scientific Systems, Inc. Submitted to: The United States Agency for International Development Under Contract No. HRN-I-00-99-00002-00, Technical Directive Number 49 August 2002 The Report of the Mid-Term Evaluation of the CHANGE Project was prepared under the auspices of the U.S. Agency for International Development (USAID) under the terms of the Monitoring, Evaluation, and Design Support (MEDS) project, Contract No. HRN-I-00-99-00002-00, Technical Directive No. 49. The opinions expressed herein are those of the authors and do not necessarily reflect the views of LTG Associates, TvT/SSS, or USAID. Information about this and other MEDS publications may be obtained from: Monitoring, Evaluation, and Design Support Project 1101 Vermont Avenue, N.W., Suite 900 Washington, DC 20005 Phone: (202) 898-0980 Fax: (202) 898-9397 jklement@medsproject.com www.medsproject.com ACRONYMS AED Academy for Educational Development AIDS Acquired immune deficiency syndrome AMARC L'Association Mondiale des Radiodiffuseurs Communautaires /World Association of Community Radio Broadcasters BASICS II Basic Support for Institutionalizing Child Survival BC Behavior Change BCC Behavior Change Communication CA Cooperating Agency CDC Centers for Disease Control and Prevention CTO Cognizant Technical Officer GAVI Global Alliance for Vaccines and Immunization HIV Human immunodeficiency virus ICRW International Center for Research on Women IEC health information, education, and communication IPCC interpersonal counseling and communication MEDS Monitoring, Evaluation and Design Support MOST USAID Micronutrient Project NGO Nongovernmental organization PAHO Pan American Health Organization PATH Program for Applied Technology in Health PVO Private voluntary organization REDSO Regional Economic Development Services Office RFA Request for Abstract RFP Request for Proposal SAMARC South Africa Medical Research Council SIGN Safe Injection Global Network SO Strategic Objective SOTA State of the Art TAG Technical Advisory Group UN United Nations USAID United States Agency for International Development WHO World Health Organization CONTENTS ACRONYMS Executive Summary .......................................................................................................... i Funding Innovation ............................................................................................................. i Selection of Innovation....................................................................................................... ii Leadership for Innovation.................................................................................................. iii Innovation .......................................................................................................................... iii Recommendations .............................................................................................................. iv CHANGE Project Management..................................................................................... iv USAID Management .......................................................................................................v Conclusion ......................................................................................................................... vi I. Introduction............ ........................................................................................................1 Objectives of the Mid-Term Evaluation ..............................................................................1 II. Findings/Issues..................... .........................................................................................4 CHANGE Project Accomplishments: Programmatic, Contractual, and Financial Objectives ............................................................................................................................4 Strengths .........................................................................................................................6 Limiting Factors .............................................................................................................6 Responsiveness of CHANGE to USAID/Washington, Missions, and other Partners to Address USAID Behavior Change Needs ...........................................................................7 Strengths .........................................................................................................................7 Weaknesses.....................................................................................................................8 Role of Innovation, Problem Solving, Capacity Building, and Application of Tools and Approaches .......................................................................................................................10 Strengths .......................................................................................................................10 Weaknesses...................................................................................................................11 Behavior Change Communication SOTA .........................................................................12 Strengths .......................................................................................................................12 Weaknesses...................................................................................................................13 III. Key Issues for the Future in BC and BCC USAID Programs ..............................15 Target Those Most in Need...........................................................................................15 Emphasize Capacity Building.......................................................................................15 Ensure Sustainability ....................................................................................................16 Systematize How to Plan and Evaluate BC Options ....................................................17 IV. Recommendations......................................................................................................18 CHANGE Project Management....................................................................................18 USAID Management ....................................................................................................19 V. Conclusion .................................................................................................................21 APPENDICES A. Tools with Potential for Further Exploitation B. Tools with Less Successful Development C. List of Contacts D. Final Scope of Work E. Tables F. Summary of Findings and Recommendations i EXECUTIVE SUMMARY A four-person team conducted a mid-term evaluation of the CHANGE project at the request of USAID from mid-March to mid-June 2002. The team employed a participatory approach throughout the information-gathering process, holding meetings and progress updates with the CHANGE co-directors, the CHANGE contractor and subcontractor, the USAID CTO, and the CHANGE team members. Evaluation team members met with key USAID officials in Washington and in the field, consulted USAID Missions by e-mail, held interviews with other key informants, and made field visits to Jamaica and the Dominican Republic. USAID designed the CHANGE project as an ambitious effort to sustain the agency’s proven and accepted leadership in behavior change (BC) and behavior change communication (BCC). The context for the project RFA is that within the agency there was a growing sense of the need for new tools and approaches in social marketing, and the need to address issues such as going to scale, sustainability of behavior change, and the knowledge–behavior gap. Evaluations had shown that BCC had lost its prominence by being mainstreamed into the Flagship projects. Outside the agency there was a feeling that more needed to be done in response to a new pluralistic media landscape in a growing number of developing countries and that a more scientific approach was needed for community-level change — sometimes referred to as “communication for social change.” During the first years of CHANGE, the project received less core funding than was necessary to carry out the comprehensive set of tasks assigned to the project. With the first work plan, the CHANGE Project and the USAID CTO reduced the number of anticipated project results to reflect the constrained core funding realities in USAID. This reduction in results was a reasonable modification that honored the spirit of the RFA. In spite of these modifications, the evaluation team felt that the CHANGE Project could achieve additional progress in selected areas in the remaining years. FUNDING INNOVATION The team found evidence that there is an important linkage between the funding streams (amounts, types, and times at which they are provided), and the ability to innovate and develop meaningful partnerships. By providing limited core funds to seed research and development activities in the early years of the Project and planning for an unrealistically high level of field support (two thirds from field support), USAID limited the potential success and benefit of CHANGE as a research and development project. Despite the handicap of limited core funds available for developing tools, approaches, and strategies, CHANGE has managed to experiment with the development of innovative approaches with a variety of partners in various settings. Some of the “innovations” have existed for some time and were modified and applied in new settings, others had been developed domestically in the United States and were applied internationally, and others are truly original. USAID research and development projects such as CHANGE need access to substantial amounts of discretionary core funding to operate successfully. To date, CHANGE has been able to ii leverage limited funds (6% of its funding) from non-US government sources; more could have provided it with more discretionary funding to use for innovation or to go to scale in certain countries.. While United States government regulations impose restrictions on using USAID money for purposes of fund raising, interpretation of these statutes should be explored to establish circumstances under which projects such as CHANGE can directly seek outside funding. In the remaining years, CHANGE should be encouraged to leverage additional outside funds. Without these funds, the project may not be able to look at issues, such as cost effectiveness, that are relatively unattractive to program implementation-driven agendas of USAID Missions. The fact that CHANGE could not identify a donor for the “cost tracking tool” is evidence of the need to provide more discretionary funding early in the project cycle. SELECTION OF INNOVATION In selecting areas for innovation, there is a need to focus on larger activities that have the potential to go to scale. The team found there is a linkage between scale (of activity size and funding) and ability to carry it out to the fullest (e.g., the Community Surveillance Kit, which was well funded through designated polio funds). The evaluation team felt that despite limited core funds, the CHANGE Project could achieve more progress in selected areas with existing funds. As a result of funding patterns, the project has been forced to undertake too many smaller activities that have stretched the ability of the CHANGE staff to monitor and provide technical support. The project has undertaken limited activities related to communications-only approaches, social marketing, and the development or evaluation of comprehensive behavior change packages. The selection of innovations would be aided by an external technical advisory group whose membership was representative of the dimensions of behavior change. Such a group could help the project keep its ear to the ground and foster experience-sharing. In this manner, creative thinkers could be brought into innovation process in a more systematic way. While these mechanisms may be costly, they are deemed a necessity and should be funded. The use of virtual technology may offer ways to reduce the costs involved. Despite the lack of such external mechanisms, CHANGE has tools under development that are promising and should be completed. It is unrealistic to expect that they will be able to do this within the remaining 13 months of the project. Several tools and models CHANGE has selected to develop are still too complex, relatively costly, and difficult to implement and replicate. Although they often show promise and in some cases have been welcomed by workers in the field, they should continue to be streamlined and simplified for easier field adaptation. CHANGE has not emphasized the communication aspects of behavior change, and has focused on the systems approach. As a result there have been some missed opportunities to advance the state of the art (SOTA) of behavior change communication. iii LEADERSHIP FOR INNOVATION The CHANGE Project has had mixed results in providing leadership for innovation. Internal management issues and staff changes have contributed to this situation. CHANGE needs to undertake a more systematic marketing effort. The Project also needs to define its role more clearly and develop a proactive approach to identifying key Missions where it can work in the remaining years. The team found little evidence that CHANGE was able to apply well-defined selection criteria regarding whether to work in a country, which has contributed to the Project behaving in a more reactive mode in its early years. Innovation requires independent leadership and strong partnerships with others. More than half (60%) of CHANGE’s partners were other USAID-funded projects, government, private sector, and academic institutions in the United States, global coalitions, multilaterals and US-based international NGOs. Less than half (40%) were Ministries of Health or Education, or southern NGOs, academic/research or private sector institutions . . In the remaining years of the Project, CHANGE should be encouraged to develop more partnerships with southern institutions, NGOs, and the private sector. In its review, the team was impressed with the talented professional staff and the fact that within the constraints of its portfolio, the CHANGE Project has provided often creative and high-quality technical assistance and has undertaken promising research and development activities. CHANGE also did an excellent job of surveying the landscape for innovation in the emerging and rapidly changing field. CHANGE organized a number of important meetings that were either regionally focused or thematic, and which have helped sustain and strengthen early innovation efforts. Had CHANGE maintained its initial intentions of holding biannual conferences, these would have had the potential to increase the SOTA of behavior change efforts. In notable cases, CHANGE proved its capacity to apply existing tools to new situations, such as with the dengue fever intervention in the Dominican Republic. In this way CHANGE has been successful in “instilling” a behavior change perspective among most of its partners and in persuading technical managers, in particular, to look through “a different lens” during strategic planning. INNOVATION The team’s review confirmed CHANGE’s early recognition that innovation in behavior change (including both communication and systems approaches) may not mean finding something that is truly new, but may nevertheless be achieved through using existing tools in new situations and adapting them.. There is an important link between the use of existing tools and capacity building, and this should be addressed by the Project. If a tool is too complex, relative to the capacity of those who will use it, it will not get used. The Project needs to concentrate more on capacity building in its remaining years, emphasizing institutional linkages. CHANGE should not only provide instruction on how to use a specific iv tool (or adapt it), but also address the enabling environment for behavior change related to human resources, which it has started doing in such areas as defining core competencies for behavior change. A particularly pressing challenge for BC and BCC, given its rapid evolution and little consensus on the most effective approaches, is sharing best practices in development and application of tools and approaches. CHANGE is well positioned to meet this challenge. RECOMMENDATIONS CHANGE Project Management The team feels strongly that the contributions of CHANGE will have lasting impact on the SOTA of BC and BCC provided that the project increase its focus and be provided with extra time in which to complete the many tasks it is facing in terms of tool development. 1. For the remaining years, the staff should focus on completing, documenting, and disseminating a small number of the promising, usable tools and on building the capacity to develop and to use behavior change interventions. There should be ample evidence that the interventions work, and the process should be streamlined to ensure that they are the least labor intensive possible and can be easily replicated by others without intensive CHANGE inputs. The team has provided an illustrative list of tools that should be pursued. The final list should be negotiated between the project CTO and the CHANGE at an upcoming benchmark meeting (see Appendix A). 2. The CHANGE Project and the CTO should establish structured benchmark (or milestone) meetings for the identification of tools and approaches for focus and development. The meeting should include senior project staff and The Manoff Group Director and review work plans and other key documents. 3. CHANGE should target countries that have relatively large amounts of child survival funding, and where innovation and new tools can make an impact on integrating behavior change into well-funded ongoing programs. By concentrating resources rather than dispersing them, CHANGE has the potential to make more lasting contributions in the area of behavior change. 4. Where possible, CHANGE should focus on developing systematic approaches at the community level and comprehensive approaches that integrate different strategies that are already known and are being applied. 5. CHANGE should add additional multidisciplinary technical expertise including an economist, behavioral scientist/social psychologist, statistician, packaging & dissemination specialists and staff members from developing countries. The Project could obtain this expertise through hiring, use of a TAG, or purchasing a percentage of the time of colleagues already working for the AED or The Manoff Group. CHANGE is already v working to improve this in the restructuring of its management team that has been in effect since June 2002 (e.g., recruitment of a dissemination specialist). 6. CHANGE should designate a senior manager or deputy director to be responsible for resource mobilization, coordination, and dissemination of results that contribute to behavioral change. 7. It is not easy to find staff with the right mix of skills to develop innovative tools as well as to build partnerships. There is a need for a separate, appropriately staffed unit within the project that can deal with partnerships. 8. CHANGE should use its connections through AED with the private sector to develop partnerships in this area. The team recognizes that success in this area will not be easy to achieve, but is worth trying over the course of the project. 9. Packaging and dissemination of results should be a high priority. CHANGE should involve those with communication expertise, including writers and information technology specialists, to ensure that the products are user friendly and include a step-by-step process guide. CHANGE should consider many vehicles for dissemination, including using the new media, as well as reviving its proposal for a biannual conference on behavior change. 10. More aggressive use of the Internet can help the CHANGE project reach out to a greater and wider audience, and also help develop a community of practice in the area of behavior change. These efforts should complement the important work already done by the Communication Initiative, and serve a more targeted audience through such means as an Extra Net (a password-protected area that people can sign up for after having registered and qualified, as opposed to an internal intranet and the public Internet). 11. CHANGE needs to better promote BC and BCC to technical health specialists who tend to hold a simplistic view that creates unrealistic expectations for what can be accomplished with the resources available. USAID Management 1. To allow the Project to complete work on existing tools and to intensify its work in areas such as community behavior change, we propose that an extension be provided of a duration that can be mutually agreed to by USAID. 2. USAID should recognize that new research and development activities might not be able to obtain significant field support until they have a proven record of success (i.e., no earlier than the third year of life). In the future, USAID should provide adequate core funding up￾front to help get these projects off the ground and on target. 3. When Missions and the global bureaus support projects such as CHANGE, there needs to be flexibility to allow the project to test out new ideas, and to evaluate and measure the effectiveness and cost effectiveness of existing approaches. In addition, USAID Missions vi should play a more catalytic role in leveraging funding from other donors, where matching funds can help extend the scale of tool development. While the “operations research” nature of the project is potentially less appealing to some Missions, it can become more attractive if it is seen as a creative contribution that USAID brings to the table in its dealings with other donors (such as the other bilaterals, the World Bank, and others). 4. USAID should also build in greater incentives and/or requirements of collaboration between and among USAID-funded activities and encourage partnership and funding from other appropriate partners who could contribute to the development and use of sustainable behavioral change tools. To implement these recommendations, it may be necessary to include language in existing cooperative agreements that requires collaboration with a project such as CHANGE. 5. USAID should ensure that key personnel include a diverse multidisciplinary team with expertise in social sciences such as sociology/anthropology, psychology, and economics as well as expertise in dissemination and diffusion of research knowledge. Technical Advisory Groups, while potentially expensive, have an important role in projects that seek to innovate and improve the SOTA. 6. USAID should place a greater emphasis on capacity building and collaboration with host￾country institutions. CONCLUSION The team is impressed with the progress that the CHANGE Project has made to date given the external constraints previously identified by the evaluation team and highlighted in the report. If USAID and CHANGE management implement the recommendations in the remaining years of the project, the Project will be even more successful. The review of the CHANGE Project’s implementation also provides important insights about the process of innovation within the organizational framework of USAID. CHANGE has demonstrated that a strong relationship exists between adequate funding streams, the ability to partner, and capacity to innovate. Within a results-focused environment in USAID Missions, a project such as CHANGE needs up￾front discretionary funding to firmly set the technical direction and pursue it over an adequate period of time. Making such a project dependent on funding from Missions has the potential to diffuse its efforts in the rightful search for resources. This can lead to missed opportunities. CHANGE was conceived and is being implemented in a purposefully competitive environment among the various cooperating agencies. To increase collaboration, there is a need for USAID to ensure that cooperating agencies (CAs) work together. Better coordination and collaboration will increase the likelihood that advances in the state of the art of behavior change can benefit a wider range of health professionals in the field. Such mandated collaboration can provide the necessary field program sites as well as add the technical and financial resources needed to adequately test vii and apply new tools and approaches, and to study the cost-effectiveness of and improve existing tools and approaches in actual field settings. CHANGE’s experience has demonstrated that innovation is relatively costly. In searching for innovation in the community and in democratic media (for example), sufficient resources are needed for credible testing and application and to create communities of interest and practice in the field of behavior change. Early efforts to study the field need to be sustained not only for the life of the project but beyond its limited scope for these efforts to have lasting results. Because these efforts involve the participation of senior practitioners, and should include a diverse range of partners, their costs may be relatively high. CHANGE has helped to better define innovation, both for identifying and developing genuinely new approaches and tools, and for adapting and fine tuning existing approaches to make them more cost effective. The more consistent application of operational definitions of innovation will have lasting effects on the implementation of future behavior change projects that are funded by USAID. Linked to the concern for under-use of existing tools and approaches may be the problem of how to build technical and programmatic capacity for behavior change and communication. Given the resource constraints of USAID-funded projects, it may be unrealistic to expect the CHANGE Project to adequately study the SOTA, develop new tools and approaches, implement them at scale, and build capacity within this limited period. CHANGE has, however, provided useful insights into some key issues in capacity building through its efforts in developing core competencies and identifying institutional mechanisms. The CHANGE Project experience to date has helped break new ground in the field of behavior change. In pursuing the recommendations proposed by the team in the remaining years of the project, CHANGE has the potential to demonstrate significant results and provide even more benefit to USAID’s health programs. The team would like to thank the CHANGE Project staff for their assistance throughout the evaluation in providing information and reacting quickly to requests for information and clarification. The team also would like to thank the USAID CTO for her helpful insights into the key questions for the evaluation. We acknowledge with thanks the technical and administrative support of the MEDS Project staff in ensuring that all our administrative needs were met. Finally, we thank the Missions and the CHANGE Project staff in the Dominican Republic and Jamaica for helping us with site visits on such short notice. viii 1 I. INTRODUCTION The purpose of this evaluation is to assess the achievements to date of the CHANGE Project and to recommend some mid-course corrections. The Project has completed 3.5 years of a planned 5- year cycle. Principal indicators of success measure the extent to which the project has been able to fulfill its proposed objectives. Following are the seven objectives of CHANGE based on the RFA, project staff discussions, and initial work plan: 1. Tools and Approaches. CHANGE will improve and expand the range and type of tools and approaches for accomplishing effective behavior change. 2. Planning and Evaluation. CHANGE will improve systems for planning and evaluating behavior change interventions. 3. Comprehensive Packages. CHANGE will demonstrate and expand the utility of behavior change packages utilizing integrated approaches to achieve normative shifts across large￾scale audiences. 4. Partnerships. CHANGE will expand the capabilities of USAID’s partners to accomplish effective behavior change. 5. Global Leadership. CHANGE will continue and expand USAID’s global leadership role in understanding and promoting the critical role of effective behavior change and developing tools to meet these goals. 6. Operations and Evaluation Research. CHANGE will expand the theory and knowledge base on behavior change, particularly with regard to cost-effectiveness, sustainability, and the ability to go to scale. 7. Capacity Building. CHANGE will expand technical expertise and technical capability within developing countries to carry out effective behavior change. OBJECTIVES OF THE MID-TERM EVALUATION The specific objectives of the CHANGE Evaluation team are as follows: 1. Review and evaluate the degree to which CHANGE has met its stated project, programmatic, contractual, and financial objectives. 2. Determine the facility of the CHANGE project to be innovative, problem solve, and apply appropriate tools and approaches. 3. Identify general options and/or lessons learned for the repositioning and redesigning of the CHANGE project in light of the environment in which it works. 2 4. In the context of objectives 1, 2, and 3, assess the ability of the CHANGE Project to respond to and consider Mission and other potential partner needs and requests. 5. Provide insights into how to best link research and development projects focusing on innovation with technical assistance projects focusing on implementation, including bilateral projects and those of Washington-based CAs. The CHANGE Mid-Term Evaluation Team is composed of three independent consultants and one Washington-based, USAID research advisor. Team members are as follows: • Moncef M. Bouhafa (team leader) — Development communication specialist, director, Center for Development Communication (U.S.-based NGO) • Neal Brandes — Child health research advisor, USAID/Global Bureau for Health • Patricia Hammer — Community participation specialist, Center for the Promotion of Social Well Being (Peruvian-based NGO) • Iain McLellan — Behavior change and communication specialist, independent consultant (Canadian based) The team employed a participatory approach throughout the information-gathering process, holding consultative meetings and progress updates with the CHANGE co-directors, the CHANGE contractor and subcontractor, the USAID CTO, and the CHANGE team members. The team held a two-day planning meeting set in motion by Jim Carney, an independent consultant specializing in meeting facilitation. Christie Billingsley clarified MEDS Project support to the team (scheduling meetings, reservation of meeting rooms, providing materials, travel logistics, administrative details, etc.). The work plan and research strategy were developed. Team members’ responsibilities for thematic areas and specific project products were designated, as well as the delegation of team tasks (e.g., contact lists, bibliography, etc.). Two sub-teams were established for site visits to Jamaica and the Dominican Republic. Questions on the CHANGE Project were developed and provided to senior management to address during the initial presentation of project advances to date. A draft work plan and time line for the evaluation were presented and approved by the CHANGE Project CTO, Elizabeth Fox. Throughout the initial two-week evaluation period in Washington, D.C., periodic team meetings were held to analyze findings and generate further questions and hypotheses. All relevant project documents were reviewed and discussed. In-depth interviews were carried out with groups and individuals. The entire team held interviews with key USAID personnel directly related to the project or its overall objectives. Individual interviews were held with key persons involved with the project: CHANGE staff, AED, Manoff Group, USAID/Washington, USAID Missions, designers of the RFA, USAID-funded CAs (MOST, Futures Group, etc.), other partners (ICRW￾International Center for Research on Women, PAHO, Save the Children, etc.). (See Appendix C for list of people interviewed.) Site visits were carried out in the third week of the evaluation in Jamaica and the Dominican Republic. In-depth interviews were held with Mission personnel and partners, including Ministry of Health officials, local NGOs, and members of the target population (community members, 3 health providers, etc.). During the field visits, the team was able to observe interactions between project staff and beneficiaries. In addition, through interviews with residents, the team was able to assess the extent to which target behaviors were being sustained. Once all data was compiled, a presentation of preliminary results was shared with the CHANGE Project staff to receive feedback, verify findings, and identify gaps or missing evidence. Following the presentation, report writing got underway, incorporating significant insights from the CHANGE staff, as well as the CTO. During May 2002, the initial draft of the report was presented to the CTO for comment, questions, and further feedback. The team incorporated suggestions and further developed the document to present as the final version of the CHANGE Mid-Term Evaluation Report the last week of May 2002. The presentation of final results to the CHANGE management team and CTO was made on June 3, 2002. 4 II. FINDINGS/ISSUES CHANGE PROJECT ACCOMPLISHMENTS: PROGRAMMATIC, CONTRACTUAL, AND FINANCIAL OBJECTIVES Several officials interviewed in USAID’s Global Bureau of Health indicated that despite the agency’s long association and leadership role in health communications, behavior change interventions and behavior change communication approaches historically have been poorly understood and under-invested components of most USAID programs. According to these and other sources, this lack of understanding and absence of consistent funding also is reflected in the ways in which investment in behavior change has been structured, either as a stand alone project, or integrated into the Flagship Technical Projects. The release of the CHANGE Project RFA in 1998 demonstrated a renewed commitment by USAID to address behavioral change and a signal to USAID Missions and the global public health community that behavior change is an integral part of successful health and development programs. The original CHANGE request for applications called for “a mechanism to assess the behavioral aspects of maternal and child health and nutrition programs and help develop and test appropriate, state-of-the-art communications, social marketing, and behavior change interventions for USAID-assisted countries.” It specifically identified the following three technical focus areas: 1. Improved tools and approaches for behavior change, communication and social marketing 2. Improved planning and evaluation tools 3. Comprehensive behavior change packages Because of difficulties in tracking and measurement, it is problematic to identify the true cost of innovation. While some tools produced by CHANGE that it feels will have lasting value have been produced when large funding streams were available, some promising tools have been less costly (e.g., the Community Surveillance Kit, which was supported by polio funds, and the work done in the Dominican Republic). The team nevertheless concludes that innovation is costly, and requires discretionary funding provided in the early years of such a project. It is no coincidence that the Community Surveillance Kit was developed over a number of years with a large amount of funding starting in the first year. How much more successful would CHANGE have been to date had it had the opportunity to similarly focus its efforts on long-term development in only a few settings, rather than deal with multiple sources of funding and requests for activities? From the beginning of the life of the Project, changing agency priorities and funding realities limited the ability of CHANGE to fully achieve its results as stated in the original proposal. The combination of insufficient core funds in the early years and many sources of funding in relatively small amounts in later years contributed to the development of a broad and diverse portfolio of relatively smallbudget activities that is difficult to manage. For example, while the CHANGE project received a grant from one source for $350,000 in its first year of operation, it received a total of $4.4 million from 22 sources in its fourth year, in amounts ranging from 5 $50,000 to $600,000. Eight sources provided less than $100,000 each and only 4 sources provided $400,000 or more (See Appendix E, Table E-4). This pattern of funding continues to challenge CHANGE’s ability to define its identity and achieve its desired impact. It would be important in the future for USAID to identify the true costs of innovation, so as to provide similar projects with appropriate financing. The CHANGE Project might be in a better position to achieve its project goals and attract more support from USAID Missions if initially it had been able to concentrate on a smaller number of host countries with a significant USAID presence. This option, however, was not possible for a number of reasons: ongoing programs in countries with other CAs, slow transition of BASICS’s II performance-based programming and development of program field sites, and CHANGE’s lack of adequate core funding. USAID could have more actively encouraged CHANGE to partner with other donors and helped leverage funding from other sources. This might have alleviated the initial deficit in core funding. The low level of initial core funding and CHANGE’s development of a diverse portfolio contributed to a fairly heavy staff workload which, in turn, contributed to consistently slow start￾ups and potential missed opportunities for synergies among CHANGE activities. These same factors also limited the ability of CHANGE to identify opportunities for greater innovation and promising behavioral interventions to evaluate, replicate, or bring to scale. The emphasis in the RFA that CHANGE activities primarily should feed into the activities of USAID flagship projects also has limited the achievement of results. The evaluation team particularly noted the instrumental role the Project CTO has played as consistent champion for the Project. She has been responsible for attracting resources and providing continuity during a transition of key personnel at the CHANGE Project. Despite her efforts, however, the lack of adequate core funds during the project’s early years limited the ability of CHANGE to undertake and achieve a more focused research and development strategy. Despite these limitations, over its 3-year life, the CHANGE Project has been a responsive USAID collaborating agency and has made progress in developing behavior change interventions and behavior change communication approaches (see Table E-5, Appendix E, for a summary of Project progress toward its intermediate results and objectives). Through its work with USAID and other partners, CHANGE has provided high-quality technical assistance and expanded the constituency for behavioral change. CHANGE has had some success in its research and development activities, an issue discussed in greater detail later in this the report. Two examples of CHANGE achievements are the work with Soul City and for GAVI and SIGN. In the case of Soul City, CHANGE helped them achieve greater impact in their “edutainment” activities through brand analysis and research; development of a “Soul Buddyz” radio series in local languages; capacity building of Soul City staff; and assistance in research. In the case of SIGN and GAVI, CHANGE applied a behavior change framework in its technical support to global advocacy efforts for injection safety and new vaccine introduction. It helped frame terms of reference and provided technical oversight in formative research with a number of other partners. 6 Although the project has a talented technical staff, it could benefit from broader disciplinary perspectives to help with the timely completion of project activities and with identifying new developments in the state of the art in behavior change and behavior change communication. Initially, the project obtained this type of expertise through an advisory group that CHANGE disbanded due to the high costs and relatively limited technical contributions of the group. Strengths • The development of the CHANGE Project demonstrates a new recognition by USAID of the importance of investing in the development and implementation of health-related behavioral change and behavioral change communication interventions. • The evaluation team was impressed by the talent and high quality of CHANGE technical staff. • Within the constraints of its portfolio, the CHANGE Project has provided often creative and high-quality technical assistance and has undertaken some promising research and development activities. Limiting Factors • USAID requirements limited the potential success and benefit of CHANGE to Washington and field Missions as a research and development activity by providing limited core funds to seed research and development activities in the early years of the Project. For example, in its first year, a third of the CHANGE budget was earmarked for work on a discrete activity related to polio. Within the constraints of its portfolio, however, CHANGE provided high-quality, often creative approaches in response to requests to undertake earmarked activities. • The emphasis on CHANGE serving the needs of flagship projects significantly delayed activities because of competing priorities, limited incentives for collaboration, and different timetables. • As a consequence of problematic funding patterns, the project has undertaken too many small activities that have stretched the ability of the CHANGE staff to monitor appropriately and provide technical support. This has led to a lack of focus. • In addition, relative to its focus on systems approaches to behavior change, the project has undertaken limited activities related to communications, social marketing, and the development or evaluation of comprehensive behavior change packages. The latter would be difficult to implement without large-scale funding, which CHANGE does not have. In similar situations, one should consider either providing more discretionary funding, or encouraging a small grants program that can provide the funding. It would be possible for the project to focus on these aspects in the remaining period. (Table E-1, Appendix E, provides a list of CHANGE communication activities). 7 • To date, neither USAID nor the CHANGE Project have sufficiently emphasized the dissemination of results or cost and cost effectiveness studies, essential for the sustainability and use of CHANGE research . RESPONSIVENESS OF CHANGE TO USAID/WASHINGTON, MISSIONS, AND OTHER PARTNERS TO ADDRESS USAID BEHAVIOR CHANGE NEEDS The original RFA and the proposal submitted by the Academy for Educational Development and The Manoff Group placed importance on partnerships. The RFA emphasized the importance of CHANGE participating in activities related to global leadership in behavior change, addressing “collaboration with other donors and technical groups.” The proposal, promised that the Project would develop a number of partnerships with “a mix of long-standing partners, and a host of fresh resourceful ones who offer USAID new skills and capabilities,” including the private sector. The team comprehensively examined the question of partnerships, looking first at those that CHANGE initiated and controlled (in a funding sense) and those where they had less flexibility in controlling the agenda. This is an important distinction since in many ways, funding limitations diverted CHANGE from its original intention to be a “think tank” that would look broadly at behavior change innovations. Given its own strategic approach that linked innovation with field realities, partnerships were and are vital to its success. (Table E-2, Appendix E, provides a complete list of CHANGE partners; Table E-6 summarizes types of partnerships by funding/activities). Strengths Project designers wanted to ensure that CHANGE was encouraged to work closely with USAID Washington, Missions, and Cooperating Agencies in developing new tools for behavior change, and in taking these to scale. This was based on the premise that tools developed in the field would be more applicable and have a better chance of being implemented at scale. In countries where CHANGE has worked on multiple activities, such as in the Dominican Republic, there is evidence that CHANGE is not only developing tools for behavior change in several fields, but also contributing to capacity building. Even in the Dominican Republic, however, funding limitations have prevented CHANGE from working at scale. CHANGE, by virtue of its nature as a relatively small and innovative project, may have been a mediating force with other cooperating agencies such as in the surveillance activity in Tanzania where CHANGE works with a USAID CA, the Partnership for Health Reform (PHR Plus), the Centers for Disease Control and Prevention (CDC), and the National Institute for Medical Research. Another area where the process of partnering is working well is CHANGE’s work defining the core competencies needed for health communication, social mobilization, and communication for social change. In this case CHANGE partners are PAHO and the Rockefeller Foundation. The objective is to develop a set of core skills for BC and BCC and in a second stage to work 8 with an institution to improve health communication training. Institutional agendas have been clearly defined, and it was clearly stated what each partner was bringing to the table. In Peru, CHANGE is working with the USAID Mission and a consortium of Peruvian universities to develop health communication capacity, focusing on these core competencies. The competency framework will have lasting benefit for the field of behavior change as a whole. “New Technologies for Disaster and Development,” the conference organized by CHANGE, is another example of partnering to position CHANGE as a global leader in behavior change communication. This type of activity helps CHANGE fulfill its global leadership role. By bringing together many partners from the private and public sectors, CHANGE has been able to catalyze a process within USAID to apply more systematic behavioral analysis and approaches to a broad range of activities. Weaknesses Although CHANGE has been responsive in terms of initiating activities to meet the needs of USAID Washington and Missions, the team felt that it could be more successful in carrying them out. Part of the problem lies with the positioning of the project in the minds of the USAID Missions. Another issue is that the USAID child survival flagship BASICS II Project was not yet sufficiently operational to provide the needed country program field sites during the first year of CHANGE program development and was not seeking behavior change technical assistance to complement their communications focus. Yet another problem lies with a less-than-vigorous effort by CHANGE itself to “market” the services and approaches it was offering. In the provision of high-quality technical assistance, the project has met or exceeded the needs of most of the Missions with which it worked. CHANGE, however, has received funding support from only seven Missions (five of which were located in the Latin America and Caribbean region) and one regional office [REDSO]. This is partly because research often is not a priority for Missions that are thinly staffed and whose first order of business is to achieve results within a set time frame. However, the evaluation team feels strongly that CHANGE could have done more to overcome this limitation by marketing and explaining itself better. Having a better mission statement and communicating CHANGE’s comparative advantages early on might have helped target certain USAID Missions for increased attention. The team found little evidence that CHANGE was able to apply well-defined selection criteria for introducing CHANGE activities into a country, contributing to the more reactive mode of the CHANGE project in its early years. Other factors also played a role, including the delayed transition of BASICS I to BASICS II, and the lack of core funding. As far as BASICS II is concerned, while both CHANGE and BASICS II Projects were designed to be technically and programmatically complementary, the partnership falls far short of what it could be, although the Projects have worked effectively together on some discrete activities, such as mother reminder materials, EPI essentials, and the materials for CORPS in Uganda. Although CHANGE contacted Mission personnel when they were visiting Washington, CHANGE could do more and should be encouraged by USAID to do a more systematic marketing effort. In the remaining years of the project, CHANGE needs to define its role more 9 clearly and develop a more proactive approach to identifying a few key Missions (with sufficient funding and interest in behavior change approaches) where it can continue to work in a significant way. In some cases, potential partners did not understand CHANGE’s role; for example, the World Association of Community Radio Broadcasters (AMARC) wanted financial in addition to technical support from CHANGE for the development of a radio self-assessment tool. While CHANGE is a technical assistance project with little core discretionary funding for field program implementation, it was misperceived by AMARC as a USAID mechanism for providing grants. As a result of this misunderstanding, CHANGE so far has been unable to achieve results in community radio, an area identified as a priority for CHANGE at the Project start. The work already done, however, has helped to develop simple audience research tools that now need to be applied in a field setting to assess their replicability and use. CHANGE has not worked as effectively as planned with other USAID flagship projects. Part of the problem may lie with the robustly competitive environment fostered by the USAID contracting process. Another issue may be that CHANGE was perceived (mainly by other CAs) as another branch of AED, one of the contractors working in the field of behavior change and BCC, rather than as an independent research and development project. Much CHANGE work is to be accomplished through partnerships with other organizations and donors. Partnerships are labor intensive, take time to develop, and require skills in negotiation. Some partners may not want to come into a process that is already started. Partnerships require full-time attention. Given its funding and staffing limitations in the first years of the project, CHANGE was not able to devote as much effort to this task as the staff felt was necessary or would have liked to devote. CHANGE has developed partnerships with 87 different groups (see Table E-2, Appendix E). More than half (60%) of CHANGE’s partners were other USAID-funded projects, government, private sector, and academic institutions in the United States, global coalitions, multilaterals and US-based international NGOs. Less than half (40%) were Ministries of Health or Education, or southern NGOs, academic/research or private sector institutions. The type and intensity of these partnerships vary widely. More than half the partnerships in the south are with implementing partners in the countries where CHANGE has developed and/or applied tools. More partnerships with developing country institutions would help broaden the Project’s focus, particularly in such areas as capacity building. The current mix of partnerships may be one of many limiting factors in “going to scale” of tools and institutionalization of the process of behavior change and behavior change communication. More staff from countries in the south might have facilitated this process. A summary of the types of partnerships by funding source is presented in Table E-6, Appendix E. In the remaining years, the Project should seek to develop more partnerships with southern institutions, NGOs, and the private sector. In general, CHANGE recognizes the need to bring a scientific approach to certain areas of behavior change (such as the community-level interventions), and to exploit knowledge from the outside (beyond USAID and strictly public health issues) in academic and “think tank” institutions. Despite its funding limitations, CHANGE has an excellent record with the 10 Rockefeller Foundation, the Technology Conference, GAVI, and ICRW. CHANGE is still somewhat limited in its effort to form partnerships with academic institutions and think tanks because of its funding and staffing constraints, with some notable exceptions. These include the partnerships with the South Africa Medical Research Council (SAMRC), the National Institute for Medical Research in Tanzania, and the Universities Consortium in Peru. CHANGE has not been able to leverage substantial funding from non-USAID sources that could have provided it with more flexibility to use for innovation or to work at scale in certain countries. Being a USAID-funded project (and thus not a legal entity) makes this process more challenging, since donors generally do not want to contribute funds to other donors, but would rather funds went directly to activities on the ground. The USAID branding of CHANGE may be a double-edged sword. On the one hand, it provides leadership and recognition, but on the other hand it makes it difficult to mobilize resources from other donors. Since USAID projects are by law required to be identified as USAID-funded, there is perhaps little that can be done about this. While there are many hurdles to overcome, it is worth noting that USAID research and development projects that rely on partnerships should have more flexibility (if possible) in attracting funding. For example, parallel funding mechanisms might be considered where CHANGE pays for certain costs, and other donors support such things as application costs and scale under an agreement negotiated at the country level by the Ministry of Health. Donor coordination mechanisms for health, while imperfect, exist in virtually every developing country and should be tried. ROLE OF INNOVATION, PROBLEM SOLVING, CAPACITY BUILDING, AND APPLICATION OF TOOLS AND APPROACHES CHANGE was given the mandate to seek innovations and examine the state of the art in the three focus areas of problem solving, capacity building, and application of tools and approaches. The RFA encouraged consideration of behavior change at the individual, community, institutional, and societal levels. The intention was to seek insights on new ways of conducting behavior change, behavior change communication, and social marketing as well as best practices to guide future planning. The team used the following definitions in its approach to the mid-term evaluation: • Innovation — developing new approaches to existing problems (as with individual behavior change) or applying existing approaches to new situations by modifying them • State of the art — identifying and sharing innovative best practices in behavior change and behavior change communication (see above) that have the potential to go to scale. Strengths Innovation CHANGE created a group of “senior advisors” who were key inputs in a series of “mini-forums” organized by CHANGE staff to review the state of the art and identify innovative tools and approaches that might be applied internationally. The list of senior advisors was impressive, and 11 a proliferation of ideas came out of these mini-forums. CHANGE did an excellent job of surveying the landscape for innovation in an emerging and rapidly changing field. In notable cases, CHANGE proved its capacity to apply innovation in that that it applied existing tools to new situations such as with the dengue control intervention in the Dominican Republic. CHANGE partners feel the Project houses substantial technical expertise in behavior change and behavior change communication. CHANGE has been successful in instilling a behavior change perspective in most of its partners and in getting technical managers to look through a different lens during strategic planning. Insisting on the use of formative research as the basis for planning and infusing top-down interventions with bottom-up inputs may not seem revolutionary. CHANGE, however, often added value to its partners with these common-sense approaches. Despite the handicap of limited core funds available for developing tools, approaches, and strategies, CHANGE managed to experiment with the development of innovative approaches with a variety of partners in various settings. Some “innovations” have existed for some time and were modified and applied in new situations, others were developed domestically in the United States and applied for the first time internationally by CHANGE, and others are truly original. Weaknesses Innovation or Adaptation? CHANGE was given the mandate to explore “innovation” as well as the SOTA. The project, however, tended to focus its attention and resources on identifying and developing “innovations.” CHANGE looked for, but still has not found, a “silver bullet” or missing piece for solving the numerous behavior change challenges. CHANGE has stated that while it started out defining innovations as “new tools and approaches,” early in the project it broadened its definition to include not only new models/tools/approaches, but also new applications of already existing tools as well as applications of existing tools by a new group of people. The team and CHANGE agree that a major difficulty with behavior change and behavior change communication is that existing tools and approaches are under-used or used poorly, not that there is a need for development of new tools and approaches. CHANGE could have done more, and in its remaining time, should do more to increase awareness of this issue. In its search for innovation, CHANGE discovered a few new models with potential applicability in non-industrialized countries. The new models had been developed in industrialized countries and many were considered by CHANGE to be too sophisticated and of limited use outside their country of origin. Part of the difficulty lies with the fact that CHANGE was limited in its search. For example, it never pursued a planned activity in its second year that would have identified and sought ideas from social scientists. This might have resulted in identifying more models that would have been applicable in developing countries. CHANGE did, however, hold periodic brown bag discussions and seminars on topics such as edutainment, early post partum care, etc. Unanticipated Demands When USAID needed public relations and information dissemination support for the UN Special Session on Children, and lacked a more suitable project mechanism to provide that support, the 12 CHANGE Project was identified for this assignment. This placed additional pressures on already diminished levels of project discretionary core funds and staff time during the critical days of project start-up. Similarly, the CHANGE mandate was adapted to support public relations and strategy work for the GAVI Initiative and to absorb significant amounts of earmarked polio funds provided to CHANGE. These examples illustrate that unanticipated funding and programming demands on CHANGE created start-up problems that only increased over time. It also illustrates that there are gaps in USAID’s portfolio of suitable mechanisms to carry out routine public relations, media relations, and public opinion activities. Too Complex Several tools and models CHANGE is developing are still too complex, making them relatively costly and difficult to implement and replicate. Though they show promise and may prove to be useful if streamlined, these tools need to be adapted further to field realities. For example, the community surveillance kit being applied in Mozambique and the resiliency and protective factors model developed in California and used by CHANGE in Jamaica need further testing, simplification, and refinement. As an example, while the kit was described as “brilliant” by a Mozambique-based PVO, they also highlighted the “need to adapt the kit or develop picture codes” given the high level of illiteracy of health volunteers. The inclusion of the adaptation guide may contribute to the perception of a “complex” package. Isolated Innovations At this stage in the project, due in part to its dependence on Mission funding and slowness in defining itself and its products, CHANGE’s efforts add up to potentially useful interventions. CHANGE should look beyond its tools, approaches, and strategies to consider what they amount to in the larger picture of behavior change and behavior change communication. CHANGE has begun working on aspects of the broader picture and in doing more it could increase the contributions made to the SOTA. CHANGE, for example, did propose a cost tracking tool, but could not identify funding. This might be an appropriate activity to pursue with its discretionary funding. BEHAVIOR CHANGE COMMUNICATION SOTA Strengths Knowledge Management Much of the work initiated by CHANGE is in progress due to the relatively slow rate of development and implementation of its tools, approaches, and strategies. BC and BCC are rapidly evolving fields with little consensus on what approaches are most effective. Therefore, knowledge management (sharing of best practices etc) is a particularly pressing challenge that CHANGE is well positioned to meet. In the remaining years, CHANGE should be able to provide useful insights into the process of developing tools and addressing why some did not 13 work and some do. For example, community-based social drama may not have worked in one country, but lessons learned might help its application in another country and setting. Disseminate Promising Tools The range of tools, approaches, and strategies being worked on by CHANGE covers a wide range of interventions. A number of promising tools, approaches, and strategies are at various states of development. Appendix A of this report provides an illustrative list. Many have potential for broader use and are worth disseminating. Tools that influence the planning process and result in the integration of behavior change strategies into larger areas are a particularly promising area. Weaknesses Missed BCC The CHANGE project was tasked with identifying tools and approaches that contribute to sustainable behavior change objectives and has emphasized a systems approach to behavior change. CHANGE acknowledges that it has not emphasized communication approaches, which it considers to have been too narrowly defined in the past. CHANGE wants to look “beyond communication” and get project designers to factor in individual and community problems and needs and think in terms of behaviors and practices rather than messages, right from the beginning. CHANGE feels that communication has too often been equated with information dissemination through the mass media rather than with behavior change problems and identifying/developing solutions. In its effort to avoid being seen “as just another communication project,” however, CHANGE appears to have missed opportunities to advance the SOTA of BCC. While CHANGE has not privileged communication activities, it nonetheless sees communication’s role as important (see Appendix E, Table E-1). This has had the unintended consequences noted in this report. The CHANGE project acknowledges it has done relatively less work in behavior change communication, per se, focusing more on systems approaches to behavior change. Research-driven, strategically planned behavior change communication has a huge arsenal of tools and approaches that it can draw on to form the common thread in a comprehensive approach. The evaluation team feels that behavior change communication can act as a central catalyst for initiating behavior change at all levels to ensure the development of effective comprehensive tools, approaches and strategies. CHANGE is encouraged to do more in this area. Dissemination CHANGE has yet to develop a dissemination strategy and structure. It launched its Web site in December 2001 and could use the new technology more to its advantage by establishing a niche among experienced behavior change and behavior change communication specialists worldwide. 14 As part of its dissemination strategy, CHANGE had originally proposed a biannual conference on behavior change. It changed its strategic focus and has decided to concentrate on smaller meetings focused on specific regions (Africa Bureau) and specific topics (e.g., technology, capacity building). These have made important contributions to the SOTA, but CHANGE should still consider holding regular conferences as a means not only of dissemination, but as a means of bringing to bear its leadership as well as that of USAID in behavior change and behavior change communication. 15 III. KEY ISSUES FOR THE FUTURE IN BC AND BCC USAID PROGRAMS The evaluation team analyzed the CHANGE Project in the context of the history of USAID BC and BCC programs over the last two decades to arrive at the following key issues for future development. It is imperative for BC and BCC programs to reach designated target populations, build capacity, ensure sustainability, systematize, and plan and evaluate interventions. Target Those Most in Need “Find an effective means of identifying high-risk populations and assisting in focusing preventive services where they are most likely to be beneficial.” [RFP] With the global recognition of the threats posed by health disparities, USAID programs increasingly will target populations with access barriers or health-defying behaviors. USAID should recognize that addressing the health needs of these populations will challenge the agency’s programs and require more effective behavior change strategies. In some cases the agency will be able to draw on successful health behavior interventions utilized in the past, although these must be appropriately modified to respond to new needs. The development of new approaches and strategies also is required. Finally, there is a need for targeted efforts that address ongoing behavioral and communications issues and mainstream behavioral approaches into field activities supported by USAID. Continual analysis and re-evaluation of state-of-the-art BC and BCC interventions developed and applied by USAID contractors, other bilaterals, and international organizations is required to ensure valuable contributions to the advancement of the discipline in both theory and practice. Emphasize Capacity Building “Strategies to do this [capacity building] might include support for study tours, state-of￾the-art meetings, and documentation and exchange of best practices.” [RFA] Capacity building in BC and BCC should be a priority for USAID at all levels of program development and operation to respond to needs of multiple stakeholders (USAID, flagship projects, practitioners in developed and developing countries). Strategic Objective (SO) leaders would benefit from orientation, information, and being kept up to date on the real and potential impact of BC and BCC interventions on morbidity and mortality. This will enhance their capabilities to comprehend the importance of BC approaches in all health programs, as well as learn how to incorporate such approaches in the development of strategic plans. Emphasis will be placed on developing capacity in BC and BCC not only with USAID Missions, but also with partners in the field. Strengthening the capacity of government institutions, non-government organizations, and civic associations, among others, to influence health behaviors holds great potential for long-term positive results in incorporating BC and BCC approaches in national strategies. Program managers in Missions and their in-country partners need appropriate skills and instruments to monitor their capacity-building efforts by carrying out baseline assessments, prioritizing targets, linking to outcomes, and measuring success. They must be provided with the 16 orientation to understand the need for behavioral approaches, as well as the process of developing country and project-specific BC and BCC interventions. This will enable them to select people to be trained, or those who have the abilities to carry out the process of conceptualizing, designing, testing, and implementing BC and BCC interventions. A key focus should be the careful and precise measurement and documentation of capacity building to establish the quality of efforts required for personnel to develop essential skills in BC and BCC. In-country capacity building requires collaboration with government and private institutions, such as ministries and universities, which create and enforce educational and training norms, standards, and certified positions in the health sector. While the area of health information, education, and communication (IEC) is standard, with corresponding curricula and technical positions in most health ministries throughout the world, a formalized, professional track for BC in the health sector should be developed. Additionally, educational and training institutes that prepare health technicians and professionals must incorporate BC skills, such as interpersonal counseling and communication (IPCC). These skills can only be practiced and sustained by the structural support of protocol and norms established at ministry levels, then carried out and supervised in hospital, clinic, and health post settings. Long-lasting changes in performance of service providers aimed at affecting clients’ health behaviors are more promising when training addresses underlying attitudes, values, and cultural norms, and provides health workers with the ability to manage organizational problems, including inadequate time, staff, and materials. Ensure Sustainability “Despite notable advances in the state-of-the-art in health communication programs, however, technical, financial, and managerial problems that compromise their effectiveness remained, notably the gap between knowledge and use of key behaviors and the fading of the behavior change when the communication supports were removed.” [RFA] How to achieve sustained BC must remain a principal endeavor in USAID efforts to improve health. Strategies should be rooted in linking policy, advocacy, and community mobilization. As discovered in the results of flagship projects, simply incorporating an IEC component into activities does little to address questions of the ability to go to scale and post-project sustainability of introduced behaviors. The slow, careful steps required to ensure sustainability must acknowledge that individual, household, and community behaviors are carried out within structures of social relations and within particular environments whose factors may inhibit or enhance sustained BC. Collective consciousness, empowerment, and action at community levels, in the form of organized groups, such as mothers’ associations or village health promoters, are fundamental to sustain introduced behaviors. BC intervention efforts must continue to explore processes of normative change that occur at the community level, and identify necessary inputs and time frames for influencing health behaviors. 17 Systematize How to Plan and Evaluate BC Options “It is important to improve planning and evaluation for selecting the best interventions to achieve project goals and for assessing costs of interventions. It will also be important to develop and apply indicators and measurement approaches to evaluate the effectiveness of alternative behavior change strategies and the costs associated with investing in them over time.” [RFA] BC programs must focus on the development of better predictive and long-term measures of BC success with specific marginalized populations (e.g., persons with HIV/AIDS, ethnic minorities, adolescent mothers). Operations research and evaluation are needed to test the cost, effectiveness, efficiency, and feasibility of particular tools and approaches to identify and resolve critical constraints on BC interventions and sustainability. 18 IV. RECOMMENDATIONS CHANGE Project Management The team feels strongly that the contributions of CHANGE will have lasting impact on the SOTA of BC and BCC provided that the project increase its focus and be provided with extra time in which to complete the many tasks it is facing in terms of tool development. 1. For the remaining years, the staff should focus on completing, documenting, and disseminating a small number of the promising, usable tools and on building the capacity to develop and to use behavior change interventions. There should be ample evidence that the interventions work, and the process should be streamlined to ensure that they are the least labor intensive possible and can be easily replicated by others without intensive CHANGE inputs. The team has provided an illustrative list of tools that should be pursued. The final list should be negotiated between the project CTO and the CHANGE at an upcoming benchmark meeting (see Appendix A). 2. The CHANGE Project and the CTO should establish structured benchmark (or milestone) meetings for the identification of tools and approaches for focus and development. The meeting should include senior project staff and The Manoff Group Director and review work plans and other key documents. 3. CHANGE should target countries that have relatively large amounts of child survival funding, and where innovation and new tools can make an impact on integrating behavior change into well-funded ongoing programs. By concentrating resources rather than dispersing them, CHANGE has the potential to make more lasting contributions in the area of behavior change. 4. Where possible, CHANGE should focus on developing systematic approaches at the community level and comprehensive approaches that integrate different strategies that are already known and are being applied. 5. CHANGE should add additional multidisciplinary technical expertise including an economist, behavioral scientist/social psychologist, statistician, packaging & dissemination specialists and staff members from developing countries. The Project could obtain this expertise through hiring, use of a TAG, or purchasing a percentage of the time of colleagues already working for the AED or The Manoff Group. CHANGE is already working to improve this in the restructuring of its management team that has been in effect since June 2002 (e.g., recruitment of a dissemination specialist). 6. CHANGE should designate a senior manager or deputy director to be responsible for resource mobilization, coordination, and dissemination of results that contribute to behavioral change. 19 7. It is not easy to find staff with the right mix of skills to develop innovative tools as well as to build partnerships. There is a need for a separate, appropriately staffed unit within the project that can deal with partnerships. 8. CHANGE should use its connections through AED with the private sector to develop partnerships in this area. The team recognizes that success in this area will not be easy to achieve, but is worth trying over the course of the project. 9. Packaging and dissemination of results should be a high priority. CHANGE should involve those with communication expertise, including writers and information technology specialists, to ensure that the products are user friendly and include a step-by-step process guide. CHANGE should consider many vehicles for dissemination, including using the new media, as well as reviving its proposal for a biannual conference on behavior change. 10. More aggressive use of the Internet can help the CHANGE project reach out to a greater and wider audience, and also help develop a community of practice in the area of behavior change. These efforts should complement the important work already done by the Communication Initiative, and serve a more targeted audience through such means as an Extra Net (a password-protected area that people can sign up for after having registered and qualified, as opposed to an internal intranet and the public Internet). 11. CHANGE needs to better promote BC and BCC to technical health specialists who tend to hold a simplistic view that creates unrealistic expectations for what can be accomplished with the resources available. USAID Management 1. To allow the Project to complete work on existing tools, and to intensify its work in areas such as community behavior change, we propose that an extension be provided of a duration that can be mutually agreed to by USAID. 2. USAID should recognize that new research and development activities might not be able to obtain significant field support until they have a proven record of success (i.e., no earlier than the third year of life). In the future, USAID should provide adequate core funding up￾front to help get these projects off the ground and on target. 3. When Missions and the global bureaus support projects such as CHANGE, there needs to be flexibility to allow the project to test out new ideas, and to evaluate and measure the effectiveness and cost effectiveness of existing approaches. In addition, USAID Missions should play a more catalytic role in leveraging funding from other donors, where matching funds can help extend the scale of tool development. While the “operations research” nature of the project is potentially less appealing to some Missions, it can become more attractive if it is seen as a creative contribution that USAID brings to the table in its dealings with other donors (such as the other bilaterals, the World Bank, and others). 20 4. USAID should also build in greater incentives and/or requirements of collaboration between and among USAID-funded activities and encourage partnership and funding from other appropriate partners who could contribute to the development and use of sustainable behavioral change tools. To implement these recommendations, it may be necessary to include language in existing cooperative agreements that requires collaboration with a project such as CHANGE. 5. USAID should ensure that key personnel include a diverse multidisciplinary team with expertise in social sciences such as sociology/anthropology, psychology, and economics as well as expertise in dissemination and diffusion of research knowledge. Technical Advisory Groups, while potentially expensive, have an important role in projects that seek to innovate and improve the SOTA. 6. USAID should place a greater emphasis on capacity building and collaboration with host￾country institutions. 21 V. CONCLUSION The team is impressed with the progress that the CHANGE Project has made to date given the external constraints previously identified by the evaluation team and highlighted in the report. If USAID and the CHANGE Project management implement the recommendations in the remaining years of the project, it will be even more successful. The review of the CHANGE Project’s implementation also provides important insights about the process of innovation within the organizational framework of USAID. CHANGE has demonstrated that there is a strong relationship between adequate funding streams, the ability to partner, and capacity to innovate. Within a results-focused environment in USAID Missions, a project such as CHANGE needs up￾front discretionary funding to firmly set the technical direction and pursue it over an adequate period of time. Making such a project dependent on funding from Missions has the potential to diffuse its efforts in the rightful search for resources. This can lead to missed opportunities. CHANGE was conceived and is being implemented in a purposefully competitive environment among the various cooperating agencies. To increase collaboration, there is a need for USAID to ensure that that CAs work together. Better coordination and collaboration will increase the likelihood that advances in the state of the art of behavior change can benefit a wider range of health professionals in the field. Such mandated collaboration can provide the necessary field program sites as well as add technical and financial resources needed to adequately test, and apply new tools and approaches, and to study the cost-effectiveness of and improve existing tools and approaches in actual field settings. CHANGE’s experience has demonstrated that innovation is relatively costly. In searching for innovation in the community and in democratic media (for example), sufficient resources are needed for credible testing and application and to create communities of interest and practice in the field of behavior change. Early efforts to study the field need to be sustained not only for the life of the project but beyond its limited scope for these efforts to have lasting results. Because these efforts involve the participation of senior practitioners, and should include a diverse range of partners, their costs may be relatively high. CHANGE has helped to better define innovation, both for identifying and developing genuinely new approaches and tools, and for adapting and fine tuning existing approaches to make them more cost effective. The more consistent application of operational definitions of innovation will have lasting effects on the implementation of future behavior change projects that are funded by USAID. Linked to the concern for under-use of existing tools and approaches may be the problem of how to build technical and programmatic capacity for behavior change and communication. Given the resource constraints of USAID-funded projects, it may be unrealistic to expect the CHANGE project to adequately study the SOTA, develop new tools and approaches, implement them at scale, and build capacity within this limited period. CHANGE has, however, provided useful 22 insights into some key issues in capacity building through its efforts in developing core competencies and identifying institutional mechanisms. The CHANGE Project experience to date has helped break new ground in the field of behavior change. In pursuing the recommendations proposed by the team in the remaining years of the project, CHANGE has the potential to demonstrate significant results and provide even more benefit to USAID’s health programs. APPENDICES A. TOOLS WITH POTENTIAL FOR FURTHER EXPLOITATION B. TOOLS WITH LESS SUCCCESSFUL DEVELOPMENT C. LIST OF CONTACTS D. FINAL SCOPE OF WORK E. TABLES F. SUMMARY OF RECOMMENDATIONS AND FINDINGS APPENDIX A TOOLS WITH POTENTIAL FOR FURTHER EXPLOITATION TOOLS WITH POTENTIAL FOR FURTHER EXPLOITATION In the time that was assigned to the evaluation it was not possible to look in depth at all the tools that CHANGE had under development. The team decided to look in depth at a few tools, based on their relevance to the Strategic Objectives of USAID, their state of development, and their capacity to inform the team on cross-cutting issues such as partnership building, capacity building, etc. The results of the review are included in Appendices A and B. This list is illustrative, and the team recommends that the final list of tools that will be pursued be negotiated between the project and the CTO at a benchmark meeting. TRIALS OF IMPROVED PRACTICES Findings TIPS (Trials of Improved Practices) is a research approach used to discover which potentially efficacious behaviors are and which are not feasible and appropriate to introduce as well as to learn more about the most effective motivations and significant barriers to change. TIPS involves negotiating the householder’s choice of which behavior the household will try and then debriefing householders about the success of their trials. Although TIPS is an “old” tool (CHANGE subcontractor the Manoff Group has a 20-year + history of using TIPS, primarily related to young child feeding), CHANGE is convinced that such behavioral trials can be very helpful in any technical area and merit wide dissemination among development organizations. CHANGE used TIPS in the Dominican Republic to plan the specific, relevant content of counseling in a nutrition intervention. The tool has been also used to develop the Dominican Republic dengue intervention, and to study behaviors that will reduce indoor air pollution in South Africa. In the Dominican Republic, Ministry of Health staff using TIPS for formative research liked the results of a negotiation approach so much that they institutionalized negotiation as part of their intervention. Lessons Learned Though TIPS has proven to be useful and can be replicated in different settings and interventions, it can be made even more effective with some changes. One difficulty is the way the intervention is framed with no clear beginning and end points. It would be better if the partners created alliances and collaborated with entities and institutions from the start, which would increase the possibility of sustaining behaviors. Finally, when choosing new behaviors, it is important to take into account the context in which they will be performed. For example, health providers need to learn to negotiate behaviors since that’s not how they’re trained in a hierarchical system. Underlying themes of the transformation of power relations in health interactions must also be explicitly addressed. “Negotiation” concerns the redistribution of power in decision-making contexts. This must be thoroughly outlined, and worked through with those engaged in the application of the methodology, to sustain the negotiating process over the long term (thereby maintaining desired behaviors. Finally, a means to monitor and follow-up on introduced behaviors rather than measuring the acquisition, maintenance, or diffusion of newly introduced behaviors is needed. CHANGE comments that TIPS (the formative research tool) has extremely clear beginning and end points: it begins with an assessment of the household’s behavior, and ends with a debriefing of the household’s success in trying a specific behavior. They note that what may have been confusing for the evaluators is the progression from the formative research tool to a negotiation intervention in the Dominican Republic (a very exciting development, and one that CHANGE would like to encourage, but not the same thing as TIPS). They agree that a challenge in using negotiation as an intervention may be to define the end point. BEHAVE MODEL FRAMEWORK Findings The BEHAVE Model Framework was designed and piloted as a 2-3 day training in English for HIV prevention planners and then adapted and pilot-tested in Spanish to address infant feeding in Bolivia. AED has used the BEHAVE Framework with many NGOs in Central America, the Dominican Republic, and Bolivia; with community-based HIV prevention organizations in the United States; and with a number of U.S. state health departments. It has proven to be an innovative and useful BC planning tool for program managers. The behavior change approach integrates findings from four types of assessments to define priority behaviors for change, identify factors influencing these behaviors, define critical target audiences, and suggest a core set of behavior change interventions. There is also a framework for charting and analyzing assessment results into a behavior change intervention matrix. This matrix helps to clearly identify a broad range of interventions that might be required to change behaviors directly as well as to create a supportive community and policy environment for change. The model has proven itself to be useful in different settings and a variety of institutional contexts, both domestically and in developing countries. The model could be strengthened if a monitoring and evaluation element, which provides insights on how the framework has been used and modified, was added. As is generally the case, evidence of the successful use of the tool would be useful in promoting the product. EVIDENCE-BASED ADVOCACY Findings CHANGE has applied a behavior change framework in its technical support to global advocacy efforts for injection safety and new vaccine introduction. Working closely with technical colleagues at USAID, WHO, CDC, BASICS, and the Children’s Vaccine Program at PATH, CHANGE helped frame terms of reference and provided technical oversight in formative research that identified key stakeholders, clarified their ideal behaviors in support of an issue, and detailed, from their point of view, the obstacles and possible enabling factors they face. These findings were used to propose both the priority and the sequence of actions for advocacy on these two topics. CHANGE’s follow-up efforts to ensure utilization of results were also important. For injection safety, the results helped launch the Safe Injection Global Network (SIGN) and served as a basis for SIGN’s behavior change strategy. For new vaccine introduction, the study results and the very concept of a behavior change approach were centerpieces in one of the first international GAVI conferences to take place following GAVI’s creation. Lessons Learned CHANGE’s evidence-based advocacy has addressed a previously unmet need and is providing useful assistance in the design and launch of USAID initiatives. Advocacy is done more effectively when concrete research replaces guess work in the planning process. MATERNAL SURVIVAL TOOL BOX: A CONCEPTUAL FRAMEWORK Findings This tool is a “minimum package” of behavior change interventions to improve maternal survival and includes the following: Danger Signs Plus (household level), Birth Preparedness Plus (community level), and Skilled Attendance Plus (facility level). The objective is to utilize a comprehensive, multilevel approach. The assemblage of existing and newly developed tools and approaches into the “Maternal Survival Tool Box ” is not complete, but is already showing its utility. There are four different types of tools and approaches in the kit, as follows: 1. Tools that have been tested, proven successful and for which the methodology has been well-documented. CHANGE intends to update and promote their use. 2. Tools that have been tested and proven successful but for which the methodology has not been well documented. CHANGE intends to develop methodological guidelines based on program experience as well as update and promote the tools. 3. Tools and approaches that have been developed for other behaviors but have not yet been applied specifically to maternal survival. CHANGE will adapt, update, and test their applicability to maternal survival. 4. New tools, which have been neither developed nor tested. CHANGE expects to refine the concepts, then develop new tools and apply them to maternal survival. There is a one major obstacle to the application of the Maternal Survival Minimum Package Framework. Although the actual tools that go into the box are still being assembled, developed, and tested, the MS framework could be applied. However, no Mission offices or other partners have incorporated this tool into their maternal health strategy. HEALTH PROVIDER ASSESSMENT TOOL Findings The Health Provider Assessment Tool was developed in Kenya. A very thorough study was carried out to identify 97 care-giving behaviors related to maternal health care by health providers, primarily nurse midwives. Its proposed applications include as a midwife self￾assessment tool or job aid, guidelines for care, and a supervision tool. Its shortcoming is that it fails to address how applying the tool influences changes in behaviors. The only reference to possible effects of applying the assessment tool states that “The tool caused midwives to pause and reflect on their behavior and the quality of care at the facility.” No steps are indicated for how to go from “pause and reflect” to actual behavior change. The tool is considered innovative by CHANGE because if its ability to influence explicit “caring behaviors” of health providers. However, there is no consideration of how to arrive at concrete actions, and sustained BC, once the behaviors are made explicit. CHANGE seeks to build on previous work, such as the work done by the flagship Mother Care that substantiated how consciousness raising can lead to action. CHANGE now needs to clearly lay out how making implicit behaviors explicit can lead to BC and specify the required steps to take people from consciousness raising to taking actions and to sustained BC. There is a need to measure the acquisition of actual behaviors, and to monitor their sustainability over time. A monitoring and evaluation framework and follow-up system is needed to see to what extent, how, and in what circumstances particular behaviors are adapted and maintained. HIV STIGMA Findings The stigmatization of HIV/AIDS and people living with the virus and the disease is proving to be a formidable obstacle to effective programming. A research initiative investigating the causes, manifestations, and consequences of HIV/AIDS-related stigma and discriminatory acts in Ethiopia, Tanzania, and Zambia has been developed by the International Center for Research on Women in collaboration with CHANGE. The intention is to “create an understanding of the interaction between insight and action.” CHANGE’s role is to use research findings to develop pilot interventions to minimize the influence of HIV stigma on the use, and provision of prevention, care, and support programs. At present training materials on stigma are non-existent. The CHANGE deliverable will be tools and approaches. The partners want to avoid exporting models from west. One reality being faced is that sexual transmission of HIV has resulted in the transgression of existing sociosexual rules. The related challenge will be to find ways of increasing open and frank dialog about sex and sexuality. A leadership advisory council spanning sectors has been established to facilitate the process. Lessons Learned CHANGE has the opportunity of developing a product that is not only innovative but very much desired by those working in HIV/AIDS. There is evidence of good collaborative relationships between CHANGE, researchers, and partners in three countries. The participatory approach used to define issues and establish protocols contributed to that collaboration. Since there is such a pressing need for intervention, the research is being developed with the perspective of how it will apply directly to behavior change interventions from the start. CHANGE will also start experimenting with interventions as the research is developed rather than waiting until it is done. COMPETENCIES AND HEALTH COMMUNICATION TRAINING Findings CHANGE partnered with Rockefeller Foundation and the Pan American Health Organization to identify key competencies in behavior change communication. They looked at needs required for developing expertise in using both mass media and participatory approaches. CHANGE conducted a two-round Delphi survey about health communication competencies via the Communication Initiative and Iniciativa de Comunicacion Web sites. More than 300 responses were obtained in each round. Respondent demographics reflected the underlying demographics of those accessing the Web sites. More than half the respondents were from the United States and Latin America, while fewer than a tenth were francophone despite the fact that the questionnaire was available in French. The partnerships of CHANGE with Rockefeller and PAHO have worked well because there was a convergence of the different institutional agendas. Health Communication Activity focused on developing institutional capacity building in behavior change and has good potential for replication if pursued vigorously. A review of existing curricula in communication and behavior change was conducted. Lessons Learned Partnering works well when institutional agendas are clearly defined. Considering the valuable lessons to be learned from the francophone world, CHANGE should ensure that it broadens its scope to include it. The project attempted to be inclusive by using the Delphi study, for example. However, a more targeted approach identifying key informants in institutions worldwide through its partners might have provided a greater diversity of response. HEALTH COMMUNICATION CAPACITY BUILDING WITH CD-ROM Findings CHANGE supported the Beta test of an interactive CD-ROM developed by the Voice of America in collaboration with PAHO for training journalists to cover health topics. The state-of￾the-art tool replicates the reporting process and provides journalists with background information and guides them through reviewing documentation and seeking out sources. CHANGE supported the pretesting of the CD-ROM with journalists covering health issues in Jamaica. CHANGE had no technical input into the pretest process but was used as a conduit of funds from USAID to the researchers. Lessons Learned Useful improvements were made following the pretest. Journalists involved in the pretest say they are using the CD-ROM as a resource and that it resulted in an improvement in their skills. There are plans to replicate the CD-ROM in other settings. This innovative approach to training journalists allows them to work at their own speed and provides reference material for later use. RESILIENCY AND PROTECTIVE FACTORS Findings The assets-based community development intervention developed in Jamaica was based on California Healthy Kids intervention. It focuses on the challenge of adolescent reproductive health in Jamaica, where an alarmingly high teen pregnancy rate inspired action by the Ministry of Health with support from USAID. The model involves identifying what makes a difference in developing positive behaviors and explaining those protective factors to communities. This assets approach helped the USAID￾supported project “Youth Now” to develop a comprehensive approach; some of the behaviors identified are to be used to develop radio play scenarios. The model has proved to be useful for moving beyond standard focus on risky behavior. Focusing on a whole range of assets gets community leaders, school officials, and parents to examine how the whole community environment is affecting behavior choices. Regrettably, a series of delays reduced the opportunity for the research results to influence the development of the “Youth Now” project. It took a relatively long time to forge relationships with the Ministry of Health, and conflicts with researchers also delayed the process. The research identifying the important resiliency factors protecting adolescents from early and unprotected sex, cigarettes, marijuana, and alcohol use, and violent and suicidal behavior has been recently completed; the challenge being faced now is how the results will be used and how it can be generalized considering the relatively costly inputs and intense external technical assistance from CHANGE. The external inputs that were required may make replication and bringing the intervention to scale and sustaining it a challenge. The “Youth Now” CA, the Futures Group, considered there was good synergy and collaboration with CHANGE, which it credits with “getting us to look holistically at what we do.” The research results leave many unanswered questions. Futures said it “wanted a meal but got a snack.” Lessons Learned Assets-based community development interventions based on insights on resiliency and protective factors have good potential for use in other settings. The model as applied in Jamaica requires a large amount of external TA, making it relatively expensive. It is a very sophisticated intervention and because of this complexity, its utility for replication is questionable. To work, it needs to be more realistic, less sophisticated, and less expensive to use. Implementation can be expected to be problematic considering the cost of developing dialog with communities and difficulty in reaching vulnerable youths. CHANGE responds that they are troubled by this discussion of “complex tools,” which implies that developing country partners do not have the capacity to grasp complexity. Resiliency is not a particularly complex concept, and fielding the assets survey is well within the capacity of any institution that regularly runs surveys and has access to an analyst who can make and use simple scales. CHANGE believes that the problems in applying the methodology in Jamaica stemmed from a problematic research vendor, which subcontracted out the analysis, , and over-scheduled jobs that diverted their attention at key junctions. While the resulting delay did reduce the opportunity for the Youth.Now project to incorporate specific results, the early introduction of the concept of resiliency and assets-based approaches gave Youth.Now, the MOH and others a bandwagon to jump on to, providing advocacy for this comprehensive approach long before results were available. COMMUNITY SURVEILLANCE KIT Findings The purpose of the Community Surveillance Kit is to support community participation in surveillance and disease prevention, particularly for AFP/polio but also for other diseases and community concerns. The kit is intended to be used primarily by PVOs, the Peace Corps, and other groups with a community presence. Some Ministries of Health have also expressed interest in using it. The kit contents include Country Adaptation Guidelines, the Community Surveillance Coordinators Handbook, and the Community Surveillance Volunteers Handbook. The kit was drafted in August 1999, pretested in Zimbabwe and Malawi in September/October 1999, and the completed version was ready for field testing in April 2000. It has since been disseminated in various versions to PVOs, Peace Corps, WHO, UNICEF. CHANGE worked with its country partners to initiate pilot tests (particularly in Zimbabwe, Malawi, Mali, and Mozambique); these pilot projects are proceeding in Mali and Mozambique. The kit was translated into French and Portuguese in 2001. CHANGE’s plans for improving the tool include revising it to better separate out the essential from the optional sections and simplifying the proposed country planning and adaptation process. In 2001, CHANGE sent the initial version of the kit on CDs in English and French to all USAID Missions. CHANGE intends to distribute the revised kit for field testing to potential in￾country partners and to establish and maintain contact with the PVOs that initiate its use. Following monitoring visits to two or three countries it will then revise, produce, and distribute the “final” version. Lessons Learned Once the kit has been finalized, CHANGE should consider making it available in Spanish if interest is demonstrated. In the opinion of the evaluation team, the kit has potential for use in Haiti, Guinea, South East Asia, and other settings. CHANGE should collaborate with other USAID community-based disease surveillance projects that are currently underway to further exploit the tool. For example, CHANGE mentions the JHU/Save the Children SECI (sistema epidemiológico comunitario integral) in use in Bolivia since 1998; that same community surveillance system is now being adapted for use in Ghana and Nigeria by JHU/PCS with Save the Children. It would be advisable for CHANGE and PCS/SAVE to meet, share findings, and collaborate in this effort since the kit is part of the public domain and can be used by anyone. APPENDIX B TOOLS WITH LESS SUCCCESSFUL DEVELOPMENT LIST OF TOOLS WITH LESS SUCCESSFUL DEVELOPMENT TALKING REMINDERS Findings Working with BASICS and Project HOPE, with funding from Glaxo, CHANGE helped develop innovative reminder materials for use by mothers. A print material was developed and evaluated in Nicaragua; the approach is being used to develop materials in Malawi (with some technical assistance from CHANGE) and in Ghana and concomitantly to write a manual about developing such materials. The effort was a good example of collaborative partnering between government and NGOs. The development process started with the mothers’ concepts of care-seeking including barriers to timely care-seeking before looking at the medical concepts. The mothers were also shown existing materials. The innovation here is to discover and emphasize the signs that parents pay attention to, rather than start with the ones physicians note, which may not “make sense” to parents. In the course of this work on reminder materials, a Manoff staff member working with CHANGE, Marco Polo Torres, conceived of “talking reminders:” materials using small computer chips (such as those common in greeting cards in the States) that could, for example, help mothers distinguish asthma from the sounds a child with pneumonia makes. CHANGE has pursued this innovative idea to the extent of joining Project Hope and BASICS in producing a concept paper to circulate to private-sector funders. Lessons Learned The cost of reproducing innovative materials such as the “talking materials”which include recorded messages may be too high for regular programmatic use. CHANGE responds that they believe that “talking reminders” could have great potential, if the unit cost can be kept low. Experts consulted by CHANGE have advised them not to be deterred by the cost of start-up. VITAMIN A: COMMUNITY-BASED INTERVENTIONS (MOST) Findings Evidence-based operational research was developed by CHANGE for use by MOST. A sophisticated plan was developed for one African country but never implemented. There turned out to be no convergence between the plan and USAID Mission’s perceived needs. According to MOST, UNICEF perceived CHANGE as competition and opposed the CHANGE proposal. MOST defined a need for improved expertise in BC and BCC within its country staff. MOST also expected CHANGE to provide innovative state-of-the-art technical assistance and to improve the capacity of MOST partners. MOST found that CHANGE was not proactive, had no SOTA presentations, and no models to share. CHANGE staff, according to MOST, was unable to respond to their needs, in part due to staff commitments to other work. A realistic tool would ensure that BC and BCC plans were developed and implemented which could also build confidence among staff who have doubts about its utility. Lessons Learned CHANGE had an opportunity to produce useful tools and materials on how to develop and implement effective BC and BCC. CHANGE may have had trouble fulfilling its mandate of being “innovative” as well as responding to the specific BCC needs of a partner such as MOST. In the case of MOST, the state-of-the-art tools were needed in 2001 and not three years later. CHANGE comments that the attempt to partner with MOST failed for many of the reasons discussed in the Partnership section of the report. COMMUNITY RADIO Findings The purpose of this activity was to design a tool that community radio staff could use for self￾assessment and carrying out audience research. Relations between CHANGE and AMARC (World Association of Community Radios) were awkward despite the fact that AMARC was on the original list of partners in the project proposal. Although there was agreement on the need for the tool, AMARC’s and CHANGE’s different institutional agendas turned out to be irreconcilable, and a long negotiation process did not result in an activity. AMARC mistakenly perceived CHANGE as a project that could provide them with funding for their operational activities.. CHANGE wanted to use an existing network to develop an innovative approach while AMARC insisted on funding for its members as part of the process. Lessons Learned Partnering requires a clear definition of what each brings to the table early in the process. CHANGE should have worked in Community Radio at the early stages of the project given the widespread explosion of the phenomenon in developing countries and the link with community empowerment and social change. More discretionary funding for CHANGE would have allowed them to deal more effectively with AMARC. If CHANGE has a fundraising capacity, it will be able to leverage other funding for such projects from third-party donors. APPENDIX C LIST OF CONTACTS LIST OF CONTACTS CHANGE PROJECT STAFF Susan Zimicki, Co-Director CHANGE Project Anton Schneider, Co-Director CHANGE Project Mike Favin, Technical Staff CHANGE Project Rebecca Fields, Technical Staff CHANGE Project Mona Moore, Technical Staff CHANGE Project Julia Rosenbaum, Technical Staff CHANGE Project Lonna Shafritz, Technical Staff CHANGE Project Nancy Pollock, Technical Staff CHANGE Project Dana Faulkner, Former Director CHANGE Project Marcia Griffiths, President The Manoff Group Laurie Krieger, Staff Member The Manoff Group USAID Duff Gillespie, Deputy Assistant Administrator USAID/Washington Betsy Brown, Director, GH/HIDN USAID/Washington Elizabeth Fox, GH/HIDN/CS USAID/Washington Murray Trostle, Senior Public Health Advisor, GH/HIDN/CS USAID/Washington Al Bartlett, GH/HN/CS USAID/Washington Daniel Halperin, GH/HIV-AIDS USAID/Washington Steve Sposato, DCHA/PPM USAID/Washington Holly Fluty Dempsey, AFR/SD USAID/Washington Steve Landry, GH/HIDN/CS USAID/Washington Frances Davidson, GH/HIDN USAID/Washington Tim Quick, GH/HIDN/CS USAID/Washington Terry Tiffany, Acting HPN Officer USAID/Jamaica Jennifer Knight-Johnson, Project Monitor USAID/Jamaica Joyce Holfeld, Senior Policy Advisor, GH/AA USAID/Washington David Losk, PHN USAID Santo Domingo, Dominican Republic Susan Brems USAID/Bolivia Jay Andersen USAID/Bangladesh Rebecca Rohrer, Health and Population USAID/Uganda OTHER Chiro Franco, Deputy Director MOST Project Jake O’Sullivan, Program Manager MOST Project Adrienne Karechi, Communication and Behavior Change Specialist MOST Project Laura Nyblade, Researcher International Center for Research on Women Don Levy, Technical Staff The Futures Group Greg Pirio, Trainer Voice of America Maia Chung, Journalist CVMTV, Jamaica Pauline Russell Brown, Chief of Party Youth Now/The Futures Group, Jamaica Maxine Wedderburn, Research Director Hope Enterprises, Jamaica Debra Wedderburn, Research Manager Hope Enterprises, Jamaica Joyce Jarrett, Director Rural Family Support Organization, Jamaica Utealia Burrell, Manager Rural Family Support Organization, Jamaica Edith Golding, Program Officer Rural Family Support Organization, Jamaica Miguel Catalino, Director Dirección General de Promoción y Educación para la Salud (DIGPRES) Santo Domingo, DR Guillermo Gonsalvez, Vector Control Unit, Sec. of Health Santo Domingo, DR Arelis Moore de Peralta, Coord. Epidemiological Vigilance Expanded Program of Immunizations Santo Domingo, DR Renee Charleston, Nutritionist Independent Consultant Eric Swedberg, Child Survival Specialist Save the Children Alex Todd, Reproductive Health Advisor Save the Children Lisa Howard-Grabman, Senior Community Mobilization Specialist Save the Children Elli Leontsini, JHU School of Public Health Dengue Epidemiology Researcher Baltimore, MD Cristina Pedreira, Epidemiology Advisor, Immunizations Santo Domingo, DR Kelva Perez, Project Manager USAID/Santo Domingo, DR Rosa Amelia Santos, Local Coordinator CHANGE Project, Santo Domingo, DR Barney Singer, Vice President and Co-Director Academy for Educational Development, Center for Social Marketing and Behavior Change Washington, D.C. Angel Solís, Vector Control Santo Domingo, DR James A. Carney, Jr. NTL consultant for MEDS Melody Trott USAID (retired) Bill Smith, Vice President Academy for Educational Development Peggy Parlato, Vice President Academy for Educational Development Barney Singer, Vice President Academy for Educational Development Dennis Foote, Project Director Academy for Educational Development APPENDIX D FINAL SCOPE OF WORK FINAL SCOPE OF WORK The CHANGE Project Mid-Term Evaluation Via the Monitoring, Evaluation and Design/Assessment Support (MEDS) Contract Number: HRN-I-00-99-00002-00 BACKGROUND The CHANGE project is a five-year cooperative agreement between the United States Agency for International Development (USAID) and the Academy for Educational Development (AED) with The Manoff Group as subcontractor. CHANGE is USAID’s fourth generation, worldwide project, focusing on health communications and behavior change. It was designed to focus on research and development rather than technical assistance for implementation, and on widening the focus from individual behavior to change at community, organizational and policy levels. In addition, CHANGE is mandated with providing leadership in behavior change innovation and creation of state-of-the-art, multi-level strategies and tools as well as utilizing a systematic approach for selecting and evaluating behavior change interventions. Based on the RFA, project staff discussions and the initial work plan, the seven project objectives for CHANGE are: • Tools and Approaches. CHANGE will improve and expand the range and type of tools and approaches for accomplishing effective behavior change. • Planning and Evaluation. CHANGE will improve systems for planning and evaluating behavior change interventions. • Comprehensive Packages. CHANGE will demonstrate and expand the utility of behavior change packages utilizing integrated approaches to achieve normative shifts across large￾scale audiences. • Partnerships. CHANGE will expand the capabilities of USAID’s partners to accomplish effective behavior change. • Global Leadership. CHANGE will continue and expand USAID’s global leadership role in understanding and promoting the critical role of effective behavior change and developing tools to meet these goals. • Operations and Evaluation Research. CHANGE will expand the theory and knowledge base on behavior change, particularly with regard to cost-effectiveness, sustainability, and the ability to go to scale. • Capacity Building. CHANGE will expand technical expertise and technical capability within developing countries to carry out effective behavior change. The CHANGE project was awarded in September 1998 with a ceiling of approximately $22 million, one-third of which was to come from the Center for Population, Health and Nutrition of the Global Bureau and two-thirds from USAID Mission field support from other USAID Bureaus. The contract anticipated an additional $1.1 million in cost sharing. CHANGE works with a broad variety of partners: four USAID Washington strategic objective teams (maternal health, child health, infectious disease and HIV/AIDS); regional bureaus, regional offices, Missions, other USAID-funded projects - particularly the Flagship projects, other US government agencies, local and international NGOs and private and multilateral international organizations. OBJECTIVES 1. Review and evaluate the degree to which CHANGE has met its stated project programmatic, contractual and financial objectives. 2. Determine the facility of the CHANGE project to be innovative, problem solve and apply appropriate tools or approaches. 3. Identify general options and/or lessons learned for the repositioning and redesigning of the CHANGE project in light of the environment in which it works. 4. In the context of objectives 1, 2, and 3, assess the ability of the CHANGE project to respond to and consider Mission and other potential partner needs and requests. 5. Provide insights into how to best link research and development projects focusing on innovation with technical assistance projects focusing on implementation, including bilateral projects and those of Washington-based CAs. MAJOR QUESTIONS TO GUIDE THE TEAM 1. How do CHANGE activities fit the objectives of the RFA? Those stated in the proposal? In hindsight, were the RFA/proposal objectives — that is, the strong emphasis on innovation and disemphasis on routine TA and training — realistic, particularly a) as regards the interest of Missions in problem-solving and in funding innovation; b) as regards the fit of CHANGE with the Flagship CAs? 2. How has the funding pattern (source and amount) aided/hindered CHANGE in developing innovative solutions to behavior change problems? What alternative patterns should USAID consider? 3. How has the staffing pattern and staff assignment structure aided/hindered CHANGE in accomplishing its objective of developing innovative solutions? What other options should CHANGE consider? 4. How well has CHANGE been able to respond to what the Missions/other funding partners wanted/needed? 5. Has CHANGE been given the opportunity (or has CHANGE been able) to move the SOTA forward related to how to improve critical health practices? 6. Has CHANGE expanded beyond its original mandate? Has it been forward thinking? 7. Describe the unintended effects or accomplishments of the project, if any. 8. In addition, what guidance can the reviewers give to CHANGE and USAID about a) repositioning/promotion, especially to affect perceptions of potential clients? b) the need for and feasibility of evaluation, including evaluation of cost-effectiveness, of CHANGE activities. 9. A larger question, which can be partially addressed through this review is how well the combination of CHANGE with the behavior change components of the larger flagship projects in the HN portfolio has provided USAID with behavior change support, as intended? What are the constraints and promoting factors, and what could be done to improve overall behavior change support? EVALUATION STRATEGY Engage in a one or two day team planning meeting to discuss the evaluation scope of work, agree on team roles and responsibilities, clarify the evaluation expectations of USAID and CHANGE, draft an evaluation work plan and decide on methodology. The team will take a participatory approach to the evaluation: they will consult the CHANGE co￾directors, a representative of the CHANGE subcontractor and the CTO during the planning and evaluation process as much as time allows, and will communicate with them on a regular basis throughout the evaluation, providing updates at reasonable intervals. Review all relevant CHANGE documents and products: including the RFA, cooperative agreement, contractual reports, Web site, tools, etc. Interview a representative, sufficient number of key persons involved with the CHANGE project, including: CHANGE staff, key advisors at AED and Manoff, USAID/Washington, persons involved in the original design of CHANGE, USAID Missions, USAID-funded CAs (RPM Plus, BASICS II, CORE, PHR Plus, MOST, MNH), other partners (Rockefeller, Project Hope, Save the Children, other PVOs, CDC, PAHO), subcontractors (Soul City, ICRW); Make a five-day site visit to the Dominican Republic where CHANGE has engaged in three implementations. Perform analysis of information gathered. Identify project structure and operations issues, impact, expected results, unintended effects, challenges, successes and lessons learned. Make an oral presentation of findings to an audience of representatives from CHANGE and CHANGE partners, CTO and other key persons from USAID/Washington. Write a report with findings, conclusions, and recommendations. TEAM COMPOSITION Three to four team members, one being from USAID/Washington and two or three others, being independent consultants or representatives of international development partner organizations. Team members should possess an amalgam of the following attributes: knowledge of behavior change and health communications, with experience in producing mass media interventions; background study in the social sciences and public health; skills and knowledge of innovation; evaluation and design expertise; background in working with NGOs/PVOs; experience managing and/or working on USAID-funded projects; and knowledge of USAID financial funding flow. TIMELINE March to mid-April 2002 for information-gathering activities with the final report completed during April/May 2002. See illustrative timeline attached. AUDIENCE The audience for the CHANGE evaluation includes USAID and all its CHANGE partners. PRODUCTS • An evaluation work plan • An oral presentation to CHANGE partners • A thirty-page, written evaluation report presenting an executive summary, findings and recommendations. APPENDIX E TABLE E-1: CHANGE COMMUNICATION ACTIVITIES TABLE E-2: CHANGE PARTNERSHIPS TABLE E-3: CHANGE CAPACITY BUILDING/TRAINING ACTIVITIES TABLE E-4: CHANGE FUNDING BY SOURCE, ACTIVITY AND YEAR TABLE E-5: PROGRESS TOWARD CHANGE PROJECT INTERMEDIATE RESULTS AND OBJECTIVES PRESENTED IN THE FIRST WORK PLAN TABLE E-6 SUMMARY OF PARTNERSHIP TYPES BY ACTIVITIES/FUNDING SOURCES TABLE E-1: CHANGE COMMUNICATION ACTIVITIES COMMUNICATION STRATEGIES DEVELOPED • Regional communication/BC strategy for polio eradication, routine immunization, and surveillance, with WHO/AFRO • Advised on communication/BC strategy of SIGN (injection safety) • Communication strategy for introduction of quadrivalent vaccine in Mozambique • Uganda Vitamin A communication strategy • BCC Strategy for HIV/AIDS programs in Haiti, including communications and social marketing components, December 2001 • Advocacy strategy proposal for MOST • DR dengue, strategic behavior change plan and a communications strategy • Strategic plan for DR immunization, including communication component • El Salvador dengue communication strategy • El Salvador national communication strategy for emergency communication for dengue outbreak • Plan for social marketing of existing Vitamin A-supplemented products by Indian manufacturers to C and D class markets • Proposal for developing entertainment education capacity in Vietnam • Communication/BC strategy for polio eradication in Pakistan RADIO SERIES • Guided Soul Buddyz radio program in local languages – development, pretesting, evaluation MATERIALS (POSTERS, STICKERS, RADIO CAMPAIGNS, PUBLIC RELATIONS MATERIALS) • Mothers’ Reminder Materials produced and disseminated in Nicaragua and Malawi • Job aids for CORPs, including flip charts, developed in Uganda • Communication materials for U.S. Coalition for Child Survival, including Web site, 10- minute informational video, 15-page marketing booklet, and media kit • DR dengue materials (posters, stickers, radio spots) • For Jamaica Adolescent Reproductive Health, PR, and media coverage of release of study; community events creating advocacy for assets-based approach • El Salvador dengue communication materials (radio spots, posters, stickers for door-to￾door campaigns, and training video for health workers) • Mozambique introduction of quadrivalent vaccine: background materials for the press, booklet for health workers, booklet for community leaders, poster with innovative display of the immunization schedule GUIDELINES/MANUALS • Communication for Polio Eradication and Routine Immunization. Checklists and Easy Reference Guides, published by WHO, 2002 (co-written and compiled by CHANGE) • Guidelines for Developing a Mothers’ Reminder Material (Draft) • “Rapid Assessment and Response Guide” for injection safety • Extensive comments on drafts of the joint partner communication handbook for polio eradication and routine immunization and on the communication and other chapters of USAID’s Immunization Essentials, both global tools • Guidelines for training community surveillance coordinators and volunteers as part of the Community Surveillance Kit; widely disseminated in English, French, and Portuguese • Developed training module for in-service training on immunization for health workers in the Dominican Republic EVALUATIONS/ASSESSMENTS/METHODOLOGIES • Evaluation of communication/social mobilization for polio eradication in Pakistan, July 2001; (with high-level WHO and UNICEF staff) • Evaluation of Mothers’ Reminder Materials in Nicaragua • Guided Soul City Brand Analysis, including focus group discussion guide, quantitative questionnaire, quantitative research report, brand analysis TOR, and brand inventory report • Developed community radio self-evaluation indicators • Developed simple radio audience questionnaire • Methodology for five joint partner (WHO, UNICEF, USAID) country case studies on communication/social mobilization for polio eradication and routine immunization; provided the team leaders for Mozambique and Mali country studies; provided a consultant for Zambia country study • Methodology and instruments for pre-testing job aids for CORPS in Uganda TABLE E-2: CHANGE PARTNERSHIPS Partner Nature of Partnership Capacity building for Partner Activity USAID Projects Applied Research in Child Health Project (ARCH) Collaborator Developing National Strategies to Slow the Growth of Antimicrobial Resistance (AMR) Cofunder Support to Global Alliance for Vaccines and Immunization (GAVI) - Global Level Collaborator Assets-based Approaches/Positive Deviance Plus Collaborator Yes Malaria Plus-Up Collaborator Yes Mother Reminder Materials BASICS II Collaborator Presentations and facilitation of BC Meeting BASICS II/Uganda Collaborator Design of counseling materials for CHWs Calidad en Salud Projecto, Guatemala (URC) Cofunder National workshop to develop IE&C strategy for C-IMCI Environmental Health Project Collaborator World Summit for Children LearnLink Project Collaborator New Technologies in Disaster & Development Communication Meeting MEDS Project Collaborator Writing of Immunization Essentials Collaborator IVACG/INACG Collaborator Technical Assistance to the MOST Project to Improve Vitamin A Consumption MOST Project Collaborator Sustainable Distribution of Vitamin A MOST Project, Nicaragua Collaborator Strengthening Behavior Change Components of the National Micronutrient Strategy MotherCare Project Co-organizer Yes Maternal Behavior Change Conference Partners for Health Reform Plus (PHR+) Collaborator Yes Integrated Disease Surveillance System Rational Pharmaceutical Management Plus Program (RPM Plus) Collaborator Developing National Strategies to Slow the Growth of Antimicrobial Resistance (AMR) USAID bilateral Egypt Healthy Mother/Health Child Project Collaborator / Cofunder Yes Improving provider behavior Collaborator Assistance to Global Alliance for Vaccines ad Immunization (GAVI) Launch, Mozambique Collaborator Community Surveillance Kit Collaborator Increasing Immunization Coverage Health Services Delivery and Support Project/Mozambique Collaborator Assistance to Global Alliance for Vaccines ad Immunization (GAVI) Launch, Mozambique Southern NGO AMARC Speaker New Technologies in Disaster & Development Communication Meeting Conde, S.A. Subcontractor Strengthening the Expanded Program on Immunization (EPI) El Salvador local NGOs Collaborator HIV/AIDS Freeplay Speaker New Technologies in Disaster & Development Communication Meeting] Group Pivot/Mali Collaborator Yes Community Surveillance Kit Hope Enterprises, Inc Subcontractor Adolescent reproductive health (Jamaica) Local NGOs in Peru Collaborator Yes Developing Health Communication Capacity NGOs supported by USAID Hurricane reconstruction funds, El Salvador Collaborators Yes Integrated Child Health and Nutrition (AIN) Partner Nature of Partnership Capacity building for Partner Activity Rural Family Support Organization Subcontractor Yes Adolescent reproductive health (Jamaica) Collaborator New Technologies in Disaster & Development Soul City Communication Meeting Collaborator Yes Soul City: Brand Analysis; "Soul Buddyz" US-based int'l NGO Adventist Development and Relief Agency (ADRA) Collaborator World Summit for Children Alliance for the Prudent Use of Antibiotics (APUA) Collaborator Developing National Strategies to Slow the Growth of Antimicrobial Resistance (AMR) Bread for the World Collaborator World Summit for Children Collaborator Yes BC presentations at global meeting on maternal health, Guatemala CARE Collaborator Evaluation of safe motherhood project, Dinajpur Christian Children's Fund Collaborator World Summit for Children Collaborator Yes Behavior Change Tools for PVOs CORE Group (consortium of 35 NGOs) Participant World Summit for Children CORE Group Polio Project Collaborator Community Surveillance Kit Family Care International Collaborator Yes Behavior Change to Increase Skilled Attendance Freedom from Hunger, Bolivia Collaborator Exploration of the Potential for Network Marketing Global Health Council Collaborator World Summit for Children Helen Keller International Participant World Summit for Children International Center for Research on Women (ICRW) Subcontractor Research on Stigma, Discrimination & Denial in 3 African Countries and 1 Asian Country PLAN International Participant World Summit for Children Project Hope Participant World Summit for Children Project Hope (global, Nicaragua, Malawi) Collaborator/ Cofunder Yes Mother Reminder Materials Rotary International Collaborator Technical Assistance for Polio Eradication Collaborator World Summit for Children Collaborator/ cofunder Yes Community-negotiated early postpartum care Save the Children intervention Participant New Technologies in Disaster & Development Communication Meeting Save the Children/Malawi Collaborator Yes Pilot test of Diagnostic Role Play World Vision Participant World Summit for Children Worldspace Speaker New Technologies in Disaster & Development Communication Meeting Government Ministries Collaborator Yes Community-Based Control of Dengue Dominican Republic MOH Collaborator Yes Strengthening the Expanded Program on Immunization (EPI) Ghana MOH, Northern Region Collaborator Community Surveillance Kit Ghana MOH Participant Yes Mother Reminder Materials Jamaica MOH Consulted Adolescent reproductive health (Jamaica) Madagascar MOH Collaborator Assistance to Global Alliance for Vaccines ad Immunization (GAVI) Launch, Mozambique Malawi MOH Collaborator Assistance to Global Alliance for Vaccines ad Immunization (GAVI) Launch, Mozambique Malawi MOH Collaborator/ Participant Yes Mother Reminder Materials Malawi MOH (and PVOs) Collaborator Yes Community Surveillance Kit Partner Nature of Partnership Capacity building for Partner Activity Mali MOH Collaborator Yes Community Surveillance Kit Mozambique MOH Collaborator Assistance to Global Alliance for Vaccines and Immunization (GAVI) Launch and follow-up activities Mozambique Collaborator Increasing Immunization Coverage Nicaragua MOH Collaborator Yes Mother Reminder Materials Peru MOH Collaborator Yes Developing Health Communication Capacity Uganda MOH Collaborator Malaria Plus-Up Zimbabwe MOH (and PVOs) Collaborator Yes Community Surveillance Kit Peru MoE Collaborator Yes Developing Health Communication Capacity El Salvador Medical Department Collaborator HIV/AIDS El Salvador National Police Collaborator HIV/AIDS El Salvador National Prison Collaborator HIV/AIDS Global Coalition Collaborator Assistance to Global Alliance for Vaccines and Immunization (GAVI) Launch and follow-up activities Collaborator Evidence-based Advocacy for Injection Safety GAVI Collaborator Technical Assistance for Polio Eradication GAVI Advocacy Task Force on Country Coordination (TFCC) Collaborator Support to Global Alliance for Vaccines (GAVI) - Global level Safe Injection Global Network (SIGN) Collaborator Evidence-Based Advocacy for Injection Safety Multilateral International Organizations Collaborator Assistance to Global Alliance for Vaccines and Immunization (GAVI) Launch and follow-up activities UNICEF Collaborator Yes Community Surveillance Kit Collaborator Increasing Immunization Coverage Collaborator Strengthening the Expanded Program on Immunization (EPI) UNICEF World Summit for Children UNICEF HQ and Africa regional offices Collaborator Technical Assistance for Polio Eradication United Nations Fund for Population Activities (UNFPA) Collaborator Assisting UNFPA with Preparation of a Proposal Cofunder Yes Competencies for Health Communicators Collaborator CD-ROM Based Training for Health Journalists Collaborator Community-Based Control of Dengue Pan American Health Organization (PAHO) Donor Strengthening the Expanded Program on Immunization (EPI) Collaborator Increasing Immunization Coverage Co-organizer Maternal Behavior Change Conference WHO Collaborator Assistance to Global Alliance for Vaccines and Immunization (GAVI) Launch and follow-up activities Collaborator Community Surveillance Kit WHO/AFRO Collaborator Yes Technical Assistance for Polio Eradication WHO/Geneva Technical Assistance for Polio Eradication WHO & UNICEF Collaborators Evaluation of polio communication, Pakistan Partner Nature of Partnership Capacity building for Partner Activity Academic and Research Institutions Consortium of Universities: Pontifica Universidad Catolica del Peru, Universidad Peruna ayetano Heredia, Universidad del Pacifico, Universidad de Lima Collaborator Yes Developing Health Communication Capacity Miz-Hasab Research Center Contractor Yes Research on Stigma, Discrimination & Denial: Ethiopia Muhimbili Medical Center, University of Dar es Salaam Subcontractor Yes Research on Stigma, Discrimination & Denial: Tanzania National Institute for Medical Research, Tanzania Collaborator Yes Integrated Disease Surveillance System South Africa Medical Research Council Collaborator/ cofunder Yes Identifying Behavioral Interventions to Reduce Indoor Air Pollution University of the West Indies Subcontractor Adolescent reproductive health (Jamaica) ZAMBART Subcontractor Yes Research on Stigma, Discrimination & Denial: Zambia Cofunder Improving The Involvement of Social Scientists in Malaria Research ad Control Collaborator Community Surveillance Kit Collaborator Community-Based Control of Dengue Collaborator/ cofunder Yes CDCynergy Centers for Disease Control Collaborator/ Cofunder Integrated Disease Surveillance System Harvard Drug Policy Group Collaborator Developing National Strategies to Slow the Growth of Antimicrobial Resistance (AMR) Johns Hopkins University School of Public Health Collaborator World Summit for Children Johns Hopkins University/PCS Speaker New Technologies in Disaster & Development Communication Meeting London School of Hygiene and Tropical Medicine Cofunder Improving the Involvement of Social Scientists in Malaria Research ad Control Private Sector O'Brien Marketing S.A. Subcontractor Yes Soul City: Brand Analysis; "Soul Buddyz" Research International Subcontractor Yes Soul City: Brand Analysis; "Soul Buddyz" Aidmatrix Speaker New Technologies in Disaster & Development Communication Meeting AOL Speaker New Technologies in Disaster & Development Communication Meeting Hewlett Packard Speaker New Technologies in Disaster & Development Communication Meeting IBM Speaker New Technologies in Disaster & Development Communication Meeting Microsoft Speaker New Technologies in Disaster Development Communication U.S. Government Institutions US Department of Health and Human Services Participant World Summit for Children US Peace Corps Collaborator Community Surveillance Kit Voice of America (VOA) Collaborator CD-ROM Based Training for Health Journalists Partner Nature of Partnership Capacity building for Partner Activity Participant New Technologies in Disaster & Development Communication Meeting Collaborator World Summit for Children Foundations and Donors Elisabeth Glaser Pediatric AIDS Foundation World Summit for Children Grantmakers in Health World Summit for Children Overseas Development Agency of Japanese Cooperative International Assistance Donor Strengthening the Expanded Program on Immunization (EPI) Cofunder Yes Competencies for Health Communicators Rockefeller Foundation Cofunder Development Communications for the 21st Century TABLE E-3: CHANGE CAPACITY BUILDING/TRAINING ACTIVITIES WORKSHOPS ORGANIZED BY CHANGE • “Learn to BEHAVE” training, Johannesburg, South Africa, February 2002 • Mothers’ Reminder Materials Training; Malawi, April 2001(10 people from 2 countries) • Three CDCynergy Trainings in D.C. – 20 people trained; most U.S.-based, but manage people in other countries • Beta test of health journalist CD-ROM TRAINING GUIDELINES DEVELOPED • Guidelines for training community surveillance coordinators and volunteers as part of the Community Surveillance Kit, widely disseminated in English, French, and Portuguese • Training module for in-service training of most health workers in the Dominican Republic on routine immunization; served as trainer of trainers in many provinces 2001-2002 WORKSHOPS CHANGE STAFF HAVE HELPED FACILITATE • CDCynergy training – Cyprus March 2002 – 20 people from 11 countries • Research proposal development workshop, Bangkok, December 1999 – 12 people from 6 countries • Behavior change training and skill-building training, CORE Group Meeting, April 2001, 50 NGO representatives • Co-facilitated GAVI Immunization Strengthening Workshop, convened by Children’s Vaccine Program, PATH. Annecy, France, April 2001, 35-40 participants • “Learn to BEHAVE” and systematic behavior change, Winter 2000, in conjunction with NGO Networks for Health, 60 NGO participants • Disease Surveillance Roles and Responsibilities workshop in Tanzania, March 2002, 20- 25 participants PRESENTATIONS AT WORKSHOPS • Presentation at the UN Roundtable on Communications in Bahia, Brazil in November 1998 on “New Tools and Approaches for Behavior Change,” 30-35 communications and behavior change specialists from UN and other donor agencies from around the world • Presentation on expanding behavior change, CORE Group Meeting, April 2001, 150 NGO representatives • Presentation on behavior change approaches (skills-building), PAHO, Spring 2001, 50 international program directors • Presentation on trials of improved practices for PVO staff, annual CORE meeting, April 2000 • Presentation on behavior change for reproductive health, NGO Networks workshop, January 2002 • Presentation on “Rethinking Behavior Change Interventions in Health” and “Behavior Change Theories,” 50 NGO representatives, April 1999 • Presentation on advocacy for safe injection, Annual Global SIGN meeting, India, approx. 100 participants, August 2001 • Several presentations at the workshop to develop a national IE&C strategy for IMCI, Guatemala, City, February 2001 • Presentation on behavior change for disease surveillance, in conjunction with PHR, Tanzania, January 2002, 20-25 participants • Review of dengue prevention strategies, WHO/TDR, October 2000, 50 policymakers • Presentation on systematic behavior change approaches, CAs Meeting, Spring 2000, 60 program and national policymakers • Presentation on assets-based approaches/Jamaica ARH, CAs Meeting, Spring 2001, 40 CAs, program and national policymakers • Gave presentations and led discussions, BASICS BC workshop, June 18-19, 2002 • Presentation on Behavior Change for CARE Third Maternal and Neonatal Health Meeting – Guatemala, October 2000 – 40 people from 15 countries • Presentation on Social Marketing and Public Health, BEHAVE training, June 2000, June 2001 and June 2002, 60 people • Presentation on assets-based approaches and social marketing, June 2002 • Presentation on CHANGE Latin America program, focusing on NEPRAM and assets￾based approaches, Annie Casey Foundation, 15 program directors • Presentation on behavior change in reproductive health, NGO Networks for Health, January 2002 • Presentation on behavior-centered programming, BASICS II, June 2002 MEETINGS • Organized and facilitated an international meeting on lessons learned from capacity building under the polio eradication initiative, June 14, 2002 • Organized and facilitated REDSO Network meetings, ESAR, October 2001, February 2002 • Organized “Development Communications in the 21st Century” in conjunction with The Rockefeller Foundation, 2000 • Organized “Competencies for Health Communicators” in conjunction with PAHO and The Rockefeller Foundation, January 2002 • Organized and facilitated “New Technologies in Disaster and Development” Conference, conference at AED attended by over 300 participants, January 2001 • Attended Davos World Economic Forum in January 2000 as the point person (with UNICEF’s head of media) for the world-wide media coverage and press relations on Bill Gates’ Davos speech and the other related announcements about the launch of GAVI TABLE E-4: CHANGE FUNDING BY SOURCE, ACTIVITY, AND YEAR (X 1,000) Year 1 Year 2 Year 3 Year 4 Total 9/98- 10/99- 10/00- 10/01- SO3 Core (Child Survival) Administration/ tool development 650 1,000 1,040 1,012 3,702 Polio 9001,9002, 9004,9011, 9012 350 200 75 75 700 Immunization, including support for GAVI and injection safety GAVI - 9102, 9105-9108 250 200 200 650 Immunization - 9003 100 50 150 Injection Safety - 9101, 9104 140 50 50 240 Micronutrients Africa Vitamin A - 320 200 100 300 World Summit for Children - 9401 270 70 340 SO2 Maternal health - 140, 1404-1408 169 600 769 SO4 HIV/AIDS ICRW - 9202 500 400 900 Stigma (CHANGE) - 9205 85 300 385 Communication Initiative - 9201 20 20 Panos - 9204 25 25 Bellagio - 9203 100 100 SO5 Infectious diseases AMR - 9501 500 500 Surveillance - 9601 200 200 Malaria Plus Up - 9303 200 200 Regional Bureaus Africa Bureau Polio - 9011/9012 170 170 Maternal health - 1402 50 50 Malaria AFR/SD - 9312 150 150 Social Science Initiative - 9311 100 100 Asia Near East Bureau HIV/AIDS - 9206 371 371 REDSO – HIV/AIDS - 3010 125 150 275 Bureau of Humanitarian Response Private and Voluntary Cooperation - 1010 50 95 145 Policy Planning and Management - 9402 65 53 118 Missions Dominican Republic Dengue - 711 150 20 170 Immunization - 7311 250 250 Nutrition - 7312 120 107 227 SO1 Jamaica - 721 200 70 270 SO2 South Africa (Soul City) - 311 150 100 250 India Vitamin A - 511 200 200 Haiti HIV/AIDS - 741 80 80 Peru - 7510 250 250 El Salvador Dengue - 7610 600 600 HIV/AIDS - 7611 400 400 1,000 2,904 3,870 5,482 13,257 TABLE E-5: PROGRESS TOWARD CHANGE PROJECT INTERMEDIATE RESULTS (IRs) AND OBJECTIVES PRESENTED IN THE FIRST WORK PLAN IR 1 — TOOLS AND APPROACHES Improve and expand the range and type of tools and approaches for accomplishing behavior change Objective: Simple tools applied in at least six countries • Refined behavior trial methodology (DR, El Salvador, South Africa) • Job aids and pretest methodology and instruments for “community-owned resource persons” (Uganda) • Communication checklists for polio and routine immunization published by WHO (global) • Provider Behavior Assessment Tool (Egypt) • Stigma analysis tools (Zambia, Tanzania, Ethiopia) • Health journalist training CD-ROM (beta test in Jamaica) • BEHAVE approach (DR, El Salvador, Tanzania) • Revised SIGN “Rapid Assessment & Response” tool for injection safety (Mongolia) • Maternal stages of change framework (Kenya) • Diagnostic role play (Malawi, requested by another project to be applied in Bangladesh) • Framework for “A Minimum Package of Behavior Change Interventions to Improve Maternal Survival” (Kenya, Guinea, Bangladesh) • Behavioral situation analysis for disease surveillance (Tanzania) • CDCynergy beta test (Cyprus for 11 WHO EMRO countries) Ready but not applied: • Radio self-evaluation approach and draft tools (no active partner) • Approach for developing national strategies to contain AMR Objective: Medium (complex) tools applied in four countries • Assets-based approach to developing an intervention for adolescent reproductive health (Jamaica) • Evidence-based advocacy (GAVI, SIGN) • Brand Analysis applied to Soul City (South Africa) • Maternal Care Provider Behavior Assessment Tool (Kenya; Bangladesh) • Qualitative research instruments and guidelines for community — negotiation of interventions to improve utilization of early postpartum care (SAVE/Guinea) • Qualitative research instruments to investigate factors influencing use of skilled childbirth care and acceptability of “linkworkers” (FCI/Kenya) • Mothers’ reminder materials, guidelines and training curriculum (Nicaragua, Malawi) • Linked quantitative & qualitative study of barriers to immunization (Dominican Republic, Mozambique [planned]) Objective: At-scale programs implemented in two countries • Dengue (El Salvador) initiated • Community surveillance kit (Mali, Mozambique initiated) Objective: New tools and approaches developed by CHANGE documented and disseminated • Web site up December ’01 • Community Surveillance Kit CD-ROM distributed worldwide in three languages • NEPRAM presented at WHO and PAHO strategy meetings • Assets-based approach presented at National Adolescent Health Conference, Annual meeting of USAID Cooperating Agencies (2001); Save the Children Federation/AED Roundtable on Positive Deviance and other Assets-Based Approaches; Annual Social Marketing Conference (accepted for June 2002) • BEHAVE Model presented at CORE Group Annual Skills-Building meeting; incorporated into BHR/PVC Technical Resource Materials; disseminated to PVOs participating in CORE South Africa training. • AMR framework presented at Global Health Council yearly meeting • Downloadable Diagnostic role play being put on web with invitation for beta testers IR 2 — PLANNING AND EVALUATION Improve systems for the planning and evaluation of behavior change interventions. Objective: Strategic planning system developed and applied in 3 countries • Behavior change section of technical reference materials developed; workshops for CORE group • CDCynergy strategic approach to program and communication planning evaluated (Cyprus workshop for WHO EMRO countries) • Strategic planning in Haiti (HIV/AIDS) • Development of a national dengue prevention strategy (DR) • Development of strategic plan for the introduction of the new pentavalent vaccine (DR) • Development of emergency communication plan in response to dengue outbreak (El Salvador) • Behavior change analysis and strategy grid (Pakistan and Egypt, also disseminated via Checklists, SIGN meeting, Delhi 8/01) Objective: Results-based monitoring developed and applied in 3 countries • Evaluation of community-based surveillance system and recommendations for improvement (Ghana) • Improving Infectious Disease Surveillance system (Tanzania) • Study of early country-level experience with GAVI carried out in five countries (in collaboration with BASICS and GAVI Task Force on Country Coordination) • Monitoring and evaluation of Mother Reminder Materials (Nicaragua) • Community surveillance kit (application initiated in Mali and Mozambique; we have heard that it is being applied by others in Bangladesh and India; is being considered for wide implementation by CARE) Objective: At least 2 collaborations with large-scale evaluation/measurement systems carried out • Development of malaria module with DHS; being used • Development of malaria “interim survey” with DHS and other partners initiated • Evaluation of Ghana community-based surveillance system (covers 2 million people) • Improving Infectious Disease Surveillance Tanzania — initiated • Development of global immunization communications & behavior change indicators and standard country monitoring forms (with WHO, UNICEF, BASICS) IR 3 — COMPREHENSIVE PACKAGES Demonstrate and expand the ability of behavior change packages utilizing integrated approaches to achieve normative shifts across large-scale audiences Objective: Comprehensive packages implemented in 2 countries • Dengue (El Salvador) • Assisted Soul City (South Africa) • Adolescent reproductive health (Jamaica) IR 4 — PARTNERSHIPS Expand the capabilities of USAID’s partners to accomplish effective behavior change Objective: At least 3 CHANGE activities developed in partnership with global-level organizations • GAVI (advocacy, support for introduction of pentavalent in DR and quadrivalent in Mozambique) • PAHO (competencies meeting; dengue in El Salvador and DR and regional activities) • Rockefeller Foundation (communication for social change, competencies, meeting on capacity building lessons) • UNICEF (community surveillance evaluation, Ghana; dengue emergency communication campaign El Salvador; communication and behavior change) • WHO Africa Regional Office (immunization activities — five country case studies, standard country monitoring form, AFRO) • CDC (community surveillance evaluation, Ghana; dengue El Salvador and DR; CDCynergy; Tanzania IDS; Partnership for Social Science in Malaria Control) • WHO/Special Programme for Research and Training in Tropical Disease (“Dengue: Guidelines for Facilitating Sustainable Behavior Change for Dengue Prevention and Control”) • SAVE the Children (Maternal health, Guinea; Positive Deviance Plus: Implications for Applying the Positive Deviance Methodology to Technical Areas Beyond Nutrition; DRP test-pilot, Malawi) • Family Care International (Maternal health, Kenya) • Project Hope/Glaxo Wellcome (Mother reminder materials, guidelines and training curriculum) • ICRW, UNAIDS (Stigma Research in Tanzania, Ethiopia, Zambia, Vietnam) • GAVI Task Force on Country Coordination (Study of country-level perspectives) • PSRA (Surveys on new vaccine introduction) • Population Council (Maternal Health, Kenya) • Glaxo, SmithKline Beecham (Stigma Tools, Ethiopia, Zambia & Vietnam) • SIGN (BCC strategy, development and pilot use of injection safety rapid assessment and response guide) • CARE (BC presentations at global meeting on maternal health) Objective: At least 3 CHANGE activities developed in partnership with other USAID cooperating agencies • BASICS (CORPS job aids, Uganda; mother reminder materials (Nicaragua, Malawi; immunization, study of country-level perceptions of GAVI; BASICS BC meeting, June 2002) • MotherCare (meeting) • PHR Plus (Disease surveillance, Tanzania) • RPM Plus (AMR) • Futures Group International (adolescent reproductive health, Jamaica) • JSI (immunization, Mozambique, Madagascar; HM/HC project, Egypt) • CORE group (behavior change training) • SAVE the Children (assets-based approaches, maternal health, Guinea) • MEDS (Immunization Essentials) • CARE (evaluation of maternal health project, Bangladesh) • MOST (Advocacy, Zambia) • FHI, Regional BC/BCC Network in ESA • URC (meeting on maternal IE&C strategy for IMCI, Guatemala) Objective: At least 3 CHANGE activities developed in partnership with local, national or regional organizations • Soul City (children’s radio; South Africa) • Medical Research Council, South Africa (indoor air pollution) • National Institute of Medical Research, disease surveillance (Tanzania) • Regional Behavior Change Network for HIV/AIDS (Regional Center for Quality of Health Care, Kampala) • Groupe Pivot (Community surveillance system, Mali) • University of West Indies, Mona (Jamaica) Campus (assets-based approaches to adolescent reproductive health) • Rural Family Support Organization [local Jamaican NGO] (assets-based approaches to adolescent reproductive health) • Ministries of Health in numerous countries (Mozambique, Uganda, Dominican Republic) • Muhimbili Medical Center, Tanzania; ZAMBART Research, Zambia; Miz-Hasab Research Center, Ethiopia; ICDS, Vietnam (stigma research) IR 5 — GLOBAL LEADERSHIP Continue and expand USAID’s global leadership role in understanding and promoting the critical role of effective behavior change and developing tools to meet these goals Objective: CHANGE leadership role established with technical fora, interagency working groups and other experts on behavior change • May 2000 Consensus meeting on “Accelerating the Adoption of New and Underutilized Vaccines” • Bellagio meetings on Development Communication and Competencies (October 2000, January 2002) • New Technologies for Disaster and Development Communication meeting, January 2001 • Continuing participation in the Communication Initiative and in the UN Communication Roundtable • Continuing participation in the WHO/TDR Steering Committee on Strategic Social, Economic and Behavioral Research • Continuing participation in the WHO/TDR Task Force on Implementation Research • On-line discussion forum concerning evidence (being established with NGO Networks) • Consultative Meeting on Behavior Change for Child Survival in Africa (USAID Africa Bureau, Feb 2000) • Behavior Change Approaches to Indoor Air Pollution (Presentations Sep 2001 at Child Health and Environment Conference; at Global Health Council, 2002 meeting) • PAHO Regional Dengue Group (DR and El Salvador dengue) • WHO/TDR (dengue, malaria) • Presentation on Behavior Change at CARE Third Maternal and Neonatal Health Meeting (40 CARE people from 15 countries, Guatemala, 2000) • Regular (twice yearly) participation in meetings of polio partners for communication • Regular participation in GAVI advocacy Task Force, SIGN meetings, CORE meetings • MotherCare maternal health meeting (co-sponsored conference on BC in maternal health) • Davos World Economic Forum in January 2000, CHANGE Director served as point person (with UNICEF’s head of media) for the world-wide media coverage and press relations on Bill Gates’ Davos speech and the other related announcements about the launch of GAVI. Objective: Concepts, tools and approaches promoted by CHANGE recognized and accepted by key audiences • Mozambique materials (booklets for health workers and for community leaders) incorporated into Children’s Vaccine Alliance web site • Framework for a minimum package of behavior change interventions to improve maternal survival presented at WHO consultative meeting on Family and Community Practices 10/01 • Maternal Care Provider Behavior Assessment Tool pretested and accepted by key Kenyan Midwives; presented to representatives of international midwifery community Washington, D.C. • Community Surveillance Kit welcomed by MOH and PVOs • BASICS request for CHANGE to participate in Uganda CORPs work based on previous experience with mother reminder materials • El Salvador dengue funding derives from DR dengue success • WHO/CDC training for Hib rapid assessment tool incorporated ideas/concepts from CHANGE • SIGN Rapid Assessment and Response Guide revised based on input from CHANGE • Hib Vaccine advocacy paper requested and accepted by CDC, WHO, USAID • BEHAVE Framework for planning incorporated into BHR/PVC child survival grant review process; presented in the BHR/PVC Technical Resource Materials, at the CORE annual skills-building meeting • Communication for Polio Eradication and Routine Immunization, published by WHO • Presentation on “New Tools and Approaches for Behavior Change” at the UN Roundtable on Communications in Bahia, Brazil, November 1998 Objective: CHANGE products utilized by other projects and agencies and non-CHANGE activities • Community Surveillance Kit CD-Rom being considered by CARE for wide implementation • Negotiation approach diffused to filariasis in the DR (from dengue) • Maternal stages of change framework incorporated into FCI baseline KAP studies in Kenya • NEPRAM incorporated into other child survival and infectious disease control planning in the Dominican Republic IR 6 — OPERATIONS AND EVALUATION RESEARCH Expand the theory and knowledge base on behavior change, particularly with regard to cost-effectiveness, sustainability, and the ability to go to scale Objective: Tools and approaches developed by CHANGE assessed and documented with regard to feasibility, effectiveness, cost and factors affecting replicability, sustainability and scale of implementation • NEPRAM (DR) • MRM (Nicaragua) Objective: At least two significant contributions made by CHANGE to advancing the state-of￾the-art of behavior change • Refined/expanded behavior trial methodology • Evidence-based advocacy in injection safety and immunization • Applications of behavior change approach to new areas, e.g., indoor air pollution, dengue control, infectious disease surveillance, AMR • Contribution to discussion about community behavior change, social change • Expansion of positive deviance methodology to technical areas beyond nutrition IR 7 — CAPACITY BUILDING Expand the technical expertise and technical capability within developing countries to carry out effective behavior change expanded Objective: Capacity to implement and evaluate CHANGE tools and approaches developed among CHANGE implementing partners • CORE BEHAVE training — S. Africa • CDCynergy training • Mother reminder materials training in Malawi and Nicaragua (Project HOPE staff); guidelines under development • Mentoring of MRC South Africa (TIPS) • AMR (INRUD/APUA) • IDS Tanzania — formative research being applied locally for surveillance • RCQHC — BC/C Network • Community Surveillance Kit “Training Guidelines” Objective: Implementation capacity developed by CHANGE linked to mechanisms for ongoing support and reinforcement • Incorporation of behavior change guidelines in BHR/PVC planning document and criteria for evaluation • Guidelines for Developing a Mothers’ Reminder Material used by multiple HOPE country programs Objective: Capacity-building materials developed by CHANGE documented and disseminated • Community Surveillance Kit • South Africa Workshop materials for BEHAVE • Mother reminder materials guidelines and approach will be disseminated 2002 • Immunization checklists • Diagnostic role play manual (draft posted on CHANGE website with an invitation to try it and give CHANGE feedback) • Stigma Research Updates & Bulletins Table E-6: Summary of Partnership Types by Activities/Funding Sources SO3 Funding Nort h Sout h Int’l (UN) MOH/ MOE USAID SO3 CORE (Child Survival) Freedom from Hunger/Bolivia X Reminder materials X X X X Assets-based approaches X X Diagnostic role play X X Community radio tools X Beta test VOA cd-rom X X Research development – applied drug use X X X X Indoor air pollution X Assist with UNFPA proposal X Vietnam – expl potential for edutainment app X Egypt maternal health X Tech assistance for polio eradication: EPI checklists X X Polio 5 country study comm. & socmob X X X X Monitoring indicators X X Eval communication and socmob Pakistan X X Support to WHO/AFRO X Community Surveillance Kit X X X X X GAVI GAVI consensus conference X X X X Support to GAVI advocacy group X X X GAVI Mozambique X X X GAVI Madagascar X X X Injection Safety X X Micronutrients CDCynergy evaluation X X X Uganda-dev comm. plan X X Zambia X X World Summit for Children X X X (continued) Other (non-SO3) Funding Nort h Sout h Int’l (UN) MOH/ MOE USAID SO2 Maternal Health Maternal BC conference X X Guinea X CARE Evaluation X Bangladesh X X Kenya with FCI X Kenya with Pop Council X X SO4 HIV/AIDS Stigma with ICRW X X Stigma (CHANGE activity) X Communication Imitative X Panos X X Bellagio X X SO5 Infectious diseases AMR X X Surveillance X X X Malaria Plus Up X X Regional Bureaus Malaria AFR/SD – 9312 X Social Science Initiative X X X Asia Near East Bureau HIV/AIDS X X REDSO – HIV/AIDS X X Bureau of Humanitarian Response Private and Voluntary Cooperation X PPM (New Tech conference) X X X Missions Dominican Republic Dengue X X X Immunization X X X Nutrition X X Jamaica X X South Africa (Soul City) X X India vitamin A X Haiti HIV/AIDS BCC strategy X Peru X X El Salvador Dengue X X HIV/AIDS X TOTAL NUMBER 30 28 20 14 24 % OF 56 ACTIVITIES 54% 50% 36% 25% 43% APPENDIX F SUMMARY OF FINDINGS AND RECOMMENDATIONS SUMMARY OF RECOMMENDATIONS AND FINDINGS Team Finding Recommendation 1. The evaluation team particularly noted the instrumental role the Project CTO has played as consistent champion for the Project. She has been responsible for attracting resources and providing continuity during a transition of key personnel at the CHANGE Project. 1. Establish structured benchmark (or milestone) meetings to identify tools and approaches for focus and development, and review work plans and other key documents. 2. The combination of insufficient core funds in the early years contributed to the development of a broad and diverse portfolio of relatively small budget activities that is difficult to manage. A number of the tools and models CHANGE has selected to develop have proven to be overly complex, relatively costly, and difficult to implement and replicate. Although they often show promise and in some cases have been welcomed by workers in the field, they must be even more streamlined and adapted to field realities. The project has undertaken too many small activities that have stretched the ability of the CHANGE staff to monitor appropriately and provide technical support. In addition, the project has undertaken limited activities related to communications, social marketing ( in relation to its focus on systems approaches to behavior change and the development or evaluation of comprehensive behavior change packages). 2. Focus CHANGE portfolio on completing, documenting, and disseminating a small number of the promising, usable tools and on building the capacity to develop and to use behavioral change interventions. 3. To date, neither USAID nor the CHANGE Project have sufficiently emphasized the dissemination of results or cost and cost effectiveness studies, essential for the sustainability and use of CHANGE research 3. Strengthen dissemination efforts through in-house communication expertise including using the new media, as well as reviving its proposal for a biannual conference on behavior change. 4. The Project has a talented technical staff; however, it could benefit from broader disciplinary perspectives to help with timely completion of Project activities and to help identify new developments in the state of the art in behavior change and behavior change communication. Initially, the Project obtained this type of expertise through an advisory group that CHANGE disbanded due to the high costs and relatively limited technical contributions of the group 4. CHANGE should add additional multi-disciplinary technical expertise including an economist, behavioral scientist/social psychologist, statistician, packaging & dissemination specialists and staff members from developing countries. 5. CHANGE has been responsive in terms of initiating activities to meet the needs of USAID Washington and Missions. The team felt that it could be more successful in carrying them out. Part of the problem lies with the positioning of the project in the minds of the USAID Missions. CHANGE, however, has received funding support from only six Missions (five of 5. Designate a senior manager or deputy director to be responsible for resource mobilization, coordination, and dissemination of results. Team Finding Recommendation which were located in the Latin America and Caribbean region). 6. The selection of innovations would be helped by having a genuinely representative external technical advisory group that can help the Project keep its ear to the ground, and share its own experiences 6. Develop a community of practice in the area of behavior change through the stronger use of the Internet. 7. Much of the work of CHANGE is to be accomplished through partnerships with other organizations and donors. Partnerships are labor intensive, take time to develop, and require skills in negotiation. Given its funding and staffing limitations in its first years, CHANGE was not able to devote as much effort to this task as the staff themselves felt was necessary or would have liked to have been able to devote. Given its own strategic approach that linked innovation with field realities, partnerships were and are vital to its success. 7. Establish a separate unit within the project that can deal with partnerships, and is staffed accordingly. 8. CHANGE has been able to leverage limited funds from non-USAID sources, which could have provided it with more discretionary funding to use for innovation or to go to scale in certain countries. To date CHANGE has received 6 percent of its funding from other sources. 8. Develop more partnerships with the private sector. 9. CHANGE should do a more systematic marketing effort. CHANGE needs to define its role more clearly and develop a proactive approach to identifying a few key Missions where it can work in the remaining years. The CHANGE Project might have achieved its project goals more effectively and attracted more support from USAID Missions if initially it had concentrated on a smaller number of host countries with a significant USAID presence. 9. Target countries that have relatively large amounts of child survival funding, and where innovation and new tools can make an impact on integrating behavior change into well funded ongoing programs. 10. In its review, the team was impressed with the talented high quality technical CHANGE staff and the fact that within the constraints of its portfolio, the Change Project has provided often creative and high-quality technical assistance and has undertaken some promising research and development activities. 10. Increase focus, and be provided with extra time to complete the important work that the Project is currently doing. 11. The Project could focus more on capacity building in its remaining years, emphasizing the need to build institutional linkages. 11. Promote BC and BCC to technical specialists who tend to have a simplistic view that creates unrealistic expectations for what can be accomplished with the resources available. Team Finding Recommendation 12. CHANGE has rarely looked beyond its tools, approaches, and strategies to consider what they add up to in the larger picture of behavior change and behavior change communication. In its effort to avoid being seen “as just another communication project,” however, CHANGE appears to have missed opportunities to advance the SOTA of BCC. CHANGE did not define the role of communication clearly at the beginning of its mandate and has not privileged communication activities, whose role it sees as important. 12. Where possible, CHANGE should focus on developing systematic approaches at the community level and on the development of comprehensive approaches that integrate different strategies that are already known and are being applied. 13. Strengthening the capacity of government institutions, nongovernment organizations, and civic associations, among others, to influence health behaviors holds great potential for long-term positive results, in terms of the incorporation of BC and BCC approaches in national strategies 13. USAID should place a greater emphasis on capacity building and collaboration with host country institutions. 14. Much of the work initiated by CHANGE is in progress due to the relatively slow rate of development and implementation of its tools, approaches and strategies 14. USAID should provide an extension of a duration that can be mutually agreed to within USAID. 15. With limited core funds available and limited interest among some Missions for experimenting with new approaches, fewer resources than anticipated were available to conduct trials of tools and approaches, apply them in different settings for comparative purposes, and bring them to scale to evaluate cost effectiveness. CHANGE, for example, did propose a cost tracking tool, but could not identify funding. This might be an appropriate activity to pursue, with its discretionary funding. 15. Provide flexibility to allow the project to test out new ideas, evaluate, and measure the effectiveness and cost effectiveness of existing approaches. Team Finding Recommendation 16. USAID requirements limited the potential success and benefit of CHANGE to Washington and field Missions as a research and development activity by providing limited core funds to seed research and development activities in the early years of the project. In the first year of the Project, a third of the CHANGE budget was earmarked for work on a discrete activity related to polio, an area that was not a core activity for the project. It is no coincidence that the Community Surveillance Kit was developed over a number of years with a large amount of funding starting in the first year. The combination of insufficient core funds in the early years contributed to the development of a broad and diverse portfolio of relatively small budget activities that is difficult to manage. For example, while the CHANGE project received a grant from one source for US$350,000 in its first year of operation, it received a total of $4.4 million from 22 sources in its fourth year. Due to funding realities and the lack of other mechanisms, for example the need for CHANGE to support the UN Special Session on Children, at the request of USAID CHANGE ended up having to focus on interventions which were neither innovative nor SOTA and only tangentially related to BC or BCC. 16. In future USAID should provide adequate core funding up front to help get these projects off the ground and on target. 17. The emphasis on CHANGE serving the needs of flagship projects significantly delayed activities because of competing priorities, limited incentives for collaboration, and different timetables. 17. USAID should also build in greater incentives and/or requirements of collaboration between and among USAID-funded activities by including language in existing cooperative agreements that requires collaboration with a project such as CHANGE. 18. The Project has a talented technical staff; however, it could benefit from broader disciplinary perspectives to assist with the timely completion of project activities and to help identify new developments in the state of the art in behavior change and behavior change communication. 18. USAID should ensure that key personnel include a diverse multidisciplinary team with expertise in key social sciences such as sociology/anthropology, psychology, and economics as well as expertise in dissemination and diffusion of research knowledge. 19. CHANGE has not been able to leverage additional funding from non￾USAID sources that could have provided it with more flexibility to use for innovation or to work at scale in certain countries. 19. USAID Missions should play more of a catalytic role in leveraging funding from other donors, where matching funds can help extend the scale of tool development. 20. CHANGE created a group of “senior advisors” who were key inputs in a series of “mini-forums” that were organized by CHANGE staff to review the state of the art, particularly on the domestic front, and identify innovative tools and approaches that might be applied internationally. The list of senior advisors was impressive, and a proliferation of ideas came out of these mini-forums 20. USAID should consider funding Technical Advisory Groups for projects such as CHANGE which, while potentially expensive, do have an important role in projects that seek to innovate and improve the SOTA.