MID-TERM EVALUATION OF THE BALANCED PROJECT AUGUST 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Don Lauro through the Global Health Technical Assistance Project. MID-TERM EVALUATION OF THE BALANCED PROJECT DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 11-01-523) is available in printed or online versions. Online documents can be located in the GH Tech web site library at http://resources.ghtechproject.net. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00 MID-TERM EVALUATION OF THE BALANCED PROJECT i CONTENTS ACRONYMS ...........................................................................................................iii EXECUTIVE SUMMARY ....................................................................................... v I. BALANCED EVALUATION ASSIGNMENT ........................................... 1 II. METHODOLOGY FOR THE BALANCED EVALUATION ................... 3 III. OVERVIEW OF MAJOR FINDINGS OF THE EVALUATION .............. 5 Progress Toward Intermediate Results and Project Objective .................................. 5 Project Design and Structure ............................................................................................. 5 Project Strategy and Shortcomings ................................................................................... 6 Project Implementation and Improvement...................................................................... 8 IV. POSITIVE PROGRESS AND REMARKABLE RESULTS ......................11 Responding to Field Project Opportunities .................................................................. 11 Knowledge Management—Upgrading the PHE Knowledge Base and Toolkit ...... 11 Integrating FP as a PHE Intervention .............................................................................. 12 Capacity Development for Field Implementation of PHE .......................................... 12 Making Family Planning Integral to Conservation ........................................................ 12 Study Tours .......................................................................................................................... 13 South-South Collaboration and Assistance—the Role of PFPI ................................. 13 V. MISSED OPPORTUNITIES ......................................................................15 VI. CATCH-UP POSSIBILITIES ....................................................................19 VII. CONCLUSIONS ........................................................................................23 ANNEXES ANNEX A. SCOPE OF WORK ...........................................................................25 ANNEX B. DOCUMENTS REVIEWED FOR THE EVALUATION ...............33 ANNEX C. LIST OF ORGANIZATIONS AND INDIVIDUALS INTERVIEWED .................................................................................35 ANNEX D. INTERVIEWS AND FIELD VISITS ................................................39 ANNEX E. KEY INFORMANT QUESTIONNAIRE ..........................................41 ANNEX F. SYNTHESIS OF ANSWERS TO FOUR SOW QUESTIONS ......51 ii MID-TERM EVALUATION OF THE BALANCED PROJECT ANNEX G. BALANCED PROJECT MIDTERM SELF-ASSESSMENT............61 ANNEX H. BALANCED PROJECT INDICATORS AND PROGRESS TOWARD TARGETS .......................................................................75 ANNEX I. EVALUATION WORK PLAN ..........................................................79 MID-TERM EVALUATION OF THE BALANCED PROJECT iii ACRONYMS ADDO Accredited drug dispensary outlet ANE Asia and Near East AOTR Agreement Officer’s Technical Representative AWF Africa Wildlife Foundation BALANCED Building Actors and Leaders for Advancing Community Excellence in Development BMS Behavior Monitoring Survey CBD Community-based distributor CCP Center for Communication Programs CI Conservation International COMACO Community Markets for Conservation CRC Coastal Resources Center CRM Coastal resource management CT Coral Triangle CTI Coral Triangle Initiative DBC Designing for Behavior Change EAC East Africa consultant ECSP Environmental Change and Security Project EWNRA Ethio Wetlands and Natural Resources Association FHI Family Health International FP Family planning GH Global health GTZ German Technical Cooperation IEC Information, education and communication IR Intermediate result IPOPCORM Integrated Population and Coastal Resources Management JGI Jane Goodall Institute JHU Johns Hopkins University JHU-CCP Johns Hopkins University, Center for Communication Program JSI John Snow International K4Health Knowledge for Health KM Knowledge management LOOCIP Longido Community Integrated Program iv MID-TERM EVALUATION OF THE BALANCED PROJECT M&E Monitoring and evaluation MOH Ministry of Health MOU Memorandum of understanding MPA Marine protected area NRM Natural resource management OEE Office of Environment and Energy OPRH Office of Population and Reproductive Health PE Peer educator PEPFAR President’s Emergency Plan for AIDS Relief PFPI PATH Foundation Philippines Inc. PHE Population, health and environment PMP Performance monitoring plan PRB Population Reference Bureau PRH Population and reproductive health PSI Population Services International RH Reproductive health SACCO Savings and Credit Cooperative SO Strategic objective SOTA State of the art TA Technical assistance TCMP Tanzania Coastal Management Partnership TNC The Nature Conservancy TOT Training of trainers URI University of Rhode Island USAID United States Agency for International Development USAID/GH United States Agency for International Development/Global Health WCS Wildlife Conservation Society WWF World Wildlife Fund YPE Youth peer educator MID-TERM EVALUATION OF THE BALANCED PROJECT v EXECUTIVE SUMMARY This mid-term evaluation of the BALANCED Project, conducted in June and July of 2011, was funded by USAID/Population and Reproductive Health (PRH) and administratively supported by GH Tech. The evaluation consultant has extensive experience with family planning (FP) and maternal-child health, including more than 15 years of implementation experience with centrally funded and bilateral USAID projects and many years of experience supporting integrated, community-based programs. The evaluation methodology combined documentation review, key informant interviews, and site visits in Tanzania, a country in which BALANCED has had extensive on-the-ground activities. The BALANCED evaluation and this report are framed within four questions from the evaluation scope of work: 1. To what extent has the BALANCED Project achieved the expected results under each intermediate result? 2. By implementing activities under IR 1, IR 2, and IR 3, to what extent has the BALANCED Project been effective in achieving the project objective: ―Advance and support wider use of effective PHE approaches‖? 3. What are the key factors contributing to the successes and shortcomings of the project? 4. What steps should USAID and BALANCED take to address these factors (shortcomings) in the last two years of the BALANCED project? Please identify both immediate and longer term steps. The BALANCED Project is part of USAID/PRH investments to advance the integrated development approach known as population, health, and environment (PHE). This line of investments, which averages around $3 million per year, responds to a Congressional mandate that FP and other related health interventions are introduced and supported within high-density and biologically diverse areas. The project objective (PO) for BALANCED as stated in the request for application (RFA) is: Advance and support wider use of effective PHE approaches worldwide. The project is designed to achieve this PO by accomplishing three intermediate results (IRs): IR 1: Capacity built for integrated PHE implementation IR 2: PHE knowledge and tools developed, organized, synthesized, and shared IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity Awarded in September 2008, BALANCED is a five-year project, without possibility of extension. Core USAID/PRH support is $5 million over the life of the project, with potential for Mission or other USAID buy-ins up to an additional $2.5 million. Responding to the RFA, the University of Rhode Island’s Coastal Resource Center (CRC) successfully proposed Building Actors and Leaders for Advancing Community Excellence in Development, the BALANCED Project. In addition to CRC and its solid record of accomplishment in coastal resource management, particularly in Africa and Asia, two other organizations are included in this project: Conservation International (CI), a highly regarded environmental organization with a solid track record for PHE in countries such as Madagascar, the Philippines, and Mexico; and PATH Foundation Philippines, Inc.(PFPI), best known as a PHE pioneer as the result of the development and implementation of the highly successful and recently replicated Integrated Population and Coastal Resources Management (IPOPCORM) Project. vi MID-TERM EVALUATION OF THE BALANCED PROJECT To implement BALANCED, CRC took the developmental lead in knowledge management (IR 2), by organizing a PHE website (http://www.k4health.org/toolkits/phe) in close collaboration with the Johns Hopkins Center for Communications Programs. CI assumed some responsibilities for website content and communications. PFPI has had lead technical responsibility for field projects (IR 3). PFPI also makes substantial contributions to capacity building (IR 1), under direction and with support from CRC. Project administration is a shared responsibility: the director and one of two deputy directors are CRC staff members; CI provides the other deputy director. Working together and separately in accordance with needs for different activities, the three participating partners have facilitated faithful, comprehensive implementation of BALANCED. Progress has been made within each IR and along indicators related to those IRs as tracked in accordance with the performance monitoring plan (PMP). However, as staff within each organization attest and PMP indicators show, project achievements have for the most part come more slowly than expected. Some indicator targets have been relatively easy to reach, such as numbers trained, which has far surpassed its original target. In contrast, other more significant, higher-level, more-difficult-to-achieve indicators of progress, such as the number of field implementation projects and amount of funds leveraged, lag behind expectations. More difficult to measure are such higher-level aspirations as the extent to which PHE itself or USAID’s global leadership for PHE have advanced as a result of BALANCED. However, key informants outside the project interviewed as part of this evaluation were not strongly supportive that significant advances in PHE or leadership have occurred as a result of BALANCED. By project’s end—some two years from now—these and certain other significant shortfalls are likely to remain. Why progress has generally been slow and such shortfalls have occurred became the principal focus of this evaluation. BALANCED has made progress in some useful areas. For example, key informants widely reported that the website, a compendium of tools and approaches to PHE, is a vast improvement over what existed before. Though most field-level key informants reported non￾use of this one-stop information source, BALANCED has also increased the number of potential practitioners of and partners for PHE, particularly in the countries where it has been most active: the Philippines, Tanzania, Ethiopia, Ghana, and Zambia. For most, this has occurred through participation in BALANCED-sponsored training programs or workshops. The project has usefully and efficiently used these capacity-building opportunities not only to prepare for PHE implementation within specific sites, but also to expose others, particularly in East Africa, to PHE and how it is undertaken. Direct exposure to PHE and its component parts also occurred during an early study tour to the Philippines and the gold standard IPOPCORM PHE Project. Key informants who participated in the Philippines visit credited this opportunity for increasing knowledge and interest in PHE. While several key informants cited participation as memorable, the evaluator was unable to determine if it has had lasting impact. In one instance, a highly valued government official collaborating with BALANCED on Pangani field implementation was transferred soon after his return from this visit. Even so, the Philippines study tour has became a point of departure for South-South collaboration supported by BALANCED. BALANCED partners CRC and PFPI have not only provided expertise, in some cases in response to requests for technical assistance, but as recipient key informants reported, they have also been successful due to an openness to exchanging experiences with counterparts. Training participants interviewed for this evaluation were unanimous in crediting staff and consultants from both organizations for their high levels of engagement and expertise. While capacity building is one of three IRs in the project, perhaps the greatest capacity to be developed within BALANCED has occurred within PFPI itself and its ability to transfer expertise to Africa. The project has been successful in promoting South-to-South collaboration, MID-TERM EVALUATION OF THE BALANCED PROJECT vii implementing in Africa PHE approaches adapted from the PFPI model and building PFPI’s capacity as PHE technical experts. Overall, BALANCED has been somewhat slow in achieving results and has not made as much progress as expected. As one example, the project has yet to populate the website with its own tools; four guidelines remain in development and have been so for some time. Similarly slow have been efforts and success in attracting funds to BALANCED and PHE. With the exception of the Philippines, where PFPI had existing connections and experience with the USAID Mission, no funding has been directly leveraged for BALANCED from Missions serving other countries. However, a USAID/Tanzania contribution to CRC was made in 2009 to extend ongoing work within the Tanzania Coastal Management Partnership (TCMP). BALANCED, which had only recently initiated activities in Pangani when the award was made, has benefited from the additional efforts subsequently made for HIV/AIDS prevention as well as livelihoods development. An additional issue is that the project has not tapped into other funding sources. (The one exception was the buy-in to BALANCED from USAID/Asia and Near East (ANE) for activities in connection with climate change; this was dismissed by BALANCED staff as a pass-through in which they were not actively engaged.) Though economical in the sense that BALANCED has piggybacked on funding already in place, such as for environmental programs or HIV/AIDS interventions, failure to directly identify funding sources early on has affected what the project can ultimately accomplish. This shortfall hampers efforts to replicate PHE in new sites, scale up existing field projects, and have greater presence in the field. Correspondingly, indicators tracking the number of field implementation projects receiving support—let alone producing results and success stories—are behind expected targets; indicators tracking the amount of funds leveraged also show a lag. Furthermore, although BALANCED has initiated interesting experiments for PHE in countries such as Tanzania, Zambia, Ethiopia, and Ghana, these in effect have served to add family planning on to existing environmental programs more than to field a full test of PHE as a comprehensive model. An evidence base is not available for determining which of these approaches—whether an approach focused on family planning or one stressing fuller community participation—yields better results. Like the long-running debate on integrated versus vertical programs, the answer may be more related to one’s perspective than to actual evidence. At this point within BALANCED, there is a more complete fielding of PHE as an innovative model under way only in the Philippines, where the Mission provides substantial support. Building on a decade of significant amounts of donor support for PHE in the Philippines, BALANCED may contribute to documenting and understanding PHE scale up in an important way. BALANCED may make significant contributions to PHE by the project’s end. The PHE website is likely a solid platform upon which to continue accumulating tools and information on PHE. Extended under BALANCED, the Philippines may become an important proving ground for PHE scale up. A recent announcement by CRC that it is developing a short course on PHE may contribute to the development of a generation of future leaders for PHE. However, in a number of ways BALANCED has fallen short of what it could have been: field implementation supported by BALANCED is under way in only four countries; apart from the Philippines, funds have not been leveraged into BALANCED to extend and replicate PHE; and capacity for PHE has not been advanced to a notable degree. BALANCED faced a number of structural challenges. Some of these, such as an overly ambitious and underfunded design, were built into the project by USAID itself. Typical of the dilemma faced at the outset was the juxtaposition between the project objective and USAID/PRH restrictions on the use of central funds. At the highest level, BALANCED is meant to ―advance and support wider use of effective PHE approaches‖ (the addition of ―worldwide‖ to this PO viii MID-TERM EVALUATION OF THE BALANCED PROJECT apparently was a proposal add-on). For field implementation by BALANCED, however, core funding from USAID/PRH is restricted for use to only the ―P‖ (or family planning) component; at a result, from the project’s outset, fielding and advancing the PHE model in its entirety would be hamstrung. As a result, family planning and contraceptive services have been layered upon, but not truly integrated within, broader developmental and exclusively environmental contexts. There are other possible flaws or weaknesses in the design. It is not clear that well-executed knowledge management will, in fact, lead to better use of available PHE guides and tools in the field. From all reports, notwithstanding the improvement that has occurred in the project in organizing website availability of PHE materials, there is little evidence that practitioners are actually using the website for this purpose. Furthermore, presuming that Missions or other donors would be willing to build on seed grants to scale up PHE may prove wishful thinking. For one thing, with limited resources of some $20,000 per year for seed grants, it is difficult to actually field a PHE developmental model of sufficient scope to attract interest. Certainly it is not possible to do so without extensive effort to know Missions and other donors as well as to understand their strategic priorities. Missions could well have interest in using PHE to bring family planning to remote, underserved regions, but not much incentive to test the PHE model or its scale up. Furthermore, the integrated approach has always been confounded and perhaps will be forever blocked by stove-piped funding streams. Differing locations, divergent experiences, and varying responsibilities within BALANCED produced structural challenges within the project itself. For example, though three experienced partners joined for the project, none had previously worked with each other. Each partner organization was located in a different place, the farthest away headquartered in Manila. Furthermore, a sub-office of one person, the director, was created in Washington, D.C. Ostensibly created to facilitate communication with USAID/PRH and D.C.-based organizations working in PHE, this added yet another location from which BALANCED operates. (The D.C. location may have been more the prerogative of USAID/PRH, which has had a penchant for requiring D.C.-based project directors, something not necessary for a project of this size.) BALANCED compounded these configuration difficulties by not conducting an internal team building as an initial activity. Such an effort could have been most fruitfully organized around another significant and early omission widely commented on by key informants—lack of an overall vision or strategic plan to guide implementation. The evaluator has concluded, in agreement with many key informants in and outside the project, that the BALANCED Project has been deficient in strategic planning and decision-making. From inception (though not conception, as clear from strategic elements in the proposal), the project has lacked strategic direction. Although some decision-making improvements have been made over the course of three years, BALANCED still lacks an overarching strategic approach to help the team make difficult decisions in an environment of scare resources and overly ambitious goals. This deficiency underlies various shortfalls and shortcomings that have plagued, and continue to plague, implementation. While there have been some useful and even remarkable achievements, overall the project is behind where it should be at this point in time. BALANCED has not yet emerged as a leading force for PHE. PHE leaders interviewed for this evaluation do not hold BALANCED in this high regard. Some were well aware of strategic shortcomings within the project. Others more distant from BALANCED offered mixed appraisals. To an extent, as one foundation leader for PHE made explicit, this is a result of under-funding: the project design was ambitious and the amount of money at $1 million per year relatively small. Recognizing this reality, the RFA in several places emphasizes the importance of leveraging funds from other sources, both within USAID as well as beyond. That in itself—how to mobilize resources—should have been a top strategic issue for BALANCED from the beginning. MID-TERM EVALUATION OF THE BALANCED PROJECT ix However, the evaluator found no evidence that BALANCED built on its own proposal to develop an overall vision and strategic plan to launch the project. One year into the project, efforts were made to develop a more strategic approach; however, these were neither fully realized nor widely applied. Similarly, resource mobilization strategies included in Year 2 and Year 3 work plans show some effort to guide where and how BALANCED would attempt to leverage resources. Unfortunately, these relatively long lists of possibilities rarely led to concrete, positive results. Could USAID have provided better guidance on this? The RFA itself emphasizes the importance of a first year work plan (within one month of award) and a project monitoring plan with targets and indicators. This sends a signal that the project is about nuts and bolts, a checklist approach to accomplishing higher-level goals. Furthermore, linkages among the three IRs and their relation to the project objective within both the RFA and the proposal leave much unspecified. This is exactly where strategic planning would have been most helpful—to sort through possible connections and synergies, and establish priorities in terms of what needed to be done when to actually advance PHE. One high-level BALANCED key informant pointed back to USAID/PRH, saying it should have done more to help with the Missions. Alternatively, the pathway could have been smoother had a predecessor project that performed poorly in bringing together PHE resources done better in fulfilling its other mandate of informing Missions on integrated approaches such as PHE. Toward the end of the first year, USAID/PRH directly encouraged BALANCED to be more strategic in its approach and decision-making. BALANCED subsequently engaged a consultant for a strategic planning meeting and developed both vision and strategy diagrams. Once done, however, these virtually disappeared from view or use. That these were not shared—or even mentioned to the evaluator during many hours of core staff interviews and the self-assessment presentation— indicates their unimportance and non-use within the project. Lacking higher-level strategic guidance to determine which activities among many best align toward achieving the project objective, BALANCED straightforwardly defined activities, assigned responsibilities, and began implementing. Central direction has focused more on implementation details than on strategically guiding and prioritizing among a broad range of implementation possibilities. The result is a project that has opportunistically achieved some positive results, but has not yet produced progress that will cumulatively result in advancing PHE. A great deal of time has been lost. At this point, the component parts of this project will result in a whole that is not as much as designers had expected, proponents of PHE had hoped, and project implementers would like. This does not have to be. A little strategic thinking could still go a long way. It is not too late to change and improve BALANCED. A shift to higher-level strategy and some consequent reorganization are needed to make BALANCED different from what has gone before. The following are offered as specific suggestions for bringing renewed energy and spirit to the project for its final two years (these and other recommendations are provided in more detail in Section VI and throughout Annex F. Synthesis of Answers to Four SOW Questions): Strategic Planning and Management Review: Use the opportunity of this evaluation to step back and develop a strategic approach for BALANCED for its remaining years, and realign management roles and responsibilities to ensure its timely application, as needed. Develop Leaders and Leadership for PHE: BALANCED could culminate in and leave a lasting legacy for PHE by developing and implementing a leadership development course. CRC could do this, perhaps in collaboration with others, at minimal cost to the project, by applying a self-sustaining, resource-generating approach. (In August 2011, CRC announced collaborative development of a short-term course in line with this recommendation.) x MID-TERM EVALUATION OF THE BALANCED PROJECT Extend Family Planning: Given the focus on FP within field implementation, BALANCED should become more proactive in the context of USAID Mission priorities and programs for family planning. Admittedly difficult for centrally funded projects, time and attention needs to be given to learning who in the Mission to approach, and how and when to approach. Time in targeted countries should be spent not just on current implementation needs but also on identifying prospects for meeting future needs. For example, monthly family planning coordination meetings in Tanzania provide an opportunity to connect BALANCED’s ability to mobilize communities with ongoing efforts to dispatch mobile teams of providers to bring a wide range of services where they are needed. Advance Community Participation and Planning: Where and when opportunity arises, undertake a deeper approach to PHE by more fully involving communities in planning and ultimately owning interventions. Both CRC’s value-added research within BALANCED and CRC experiences in Ghana with community-mapping exercises would resonate with such an implementation opportunity. Working more community participation and planning into the PFPI model would be a useful addition. Brokering Funds and Backloading Seed Grants: Given project time remaining and the small window for Mission buy-ins, BALANCED needs to change tactics. Rather than focusing on buy-ins, the project should assist organizations with which it is working, such as TCMP, in attracting funds to continue and expand PHE into the future beyond BALANCED. If capacity has been built for PHE, this should be something that these organizations can manage. Similarly, for seed grants being considered by BALANCED, larger amounts of funding than past awards may be warranted for and well used by some of these organizations in the time remaining. MID-TERM EVALUATION OF THE BALANCED PROJECT 1 I. BALANCED EVALUATION ASSIGNMENT At the request of USAID/Population and Reproductive Health (PRH), GH Tech identified a qualified consultant in May 2011 to conduct a mid-term evaluation of its flagship project for population, health, and environment (PHE), Building Actors and Leaders for Advancing Community Excellence in Development (BALANCED). This evaluation took place from June to August 2011, just after the midway point of the BALANCED Project. The BALANCED Evaluation scope of work (SOW), found in Annex A, specified that interviews of key informants and field site visits within a specified country in which project implementation is under way would serve as the primary means for undertaking this evaluation. Accordingly, the consultant developed an evaluation methodology (overviewed in Section II), prepared for a trip to Tanzania from June 24 to July 10, and submitted an evaluation work plan (see Annex I) to USAID, which was approved on June 29, 2011. The five-year cooperative agreement, awarded to the University of Rhode Island’s Coastal Resource Center (CRC) in September 2008, oriented BALANCED toward the primary project objective (PO): Advance and support wider use of effective PHE approaches. Together with partnering organizations Conservation International (CI) and PATH Foundation Philippines, Inc. (PFPI), CRC organized the project to advance three related intermediate results (IRs): IR 1—Capacity built for integrated PHE implementation IR 2—PHE knowledge and tools developed, organized, synthesized, and shared IR 3—Results-oriented PHE field activities implemented in areas of high biodiversity Corresponding to the PO and these IRs, the evaluator interviewed more than 40 key informants and visited eight field sites in Tanzania to explore project strengths as well as shortcomings. As specified in the SOW, this evaluation was guided by seven principal questions: four focused on BALANCED itself (70% of the evaluation level of effort [LOE]) and three on future directions for USAID/PRH in PHE (30% LOE). This report concentrates on BALANCED itself and these first four SOW questions. For proprietary reasons, a separate confidential memo has been prepared for USAID organized around the three future-oriented questions. 2 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 3 II. METHODOLOGY FOR THE BALANCED EVALUATION Guided by the SOW, the evaluator developed the evaluation methodology to be used early in the assignment; a description of the methodology is provided in the evaluation work plan, submitted on June 20. As expected, some methodological modifications occurred during the course of the evaluation. For example, as the evaluator learned more about the project and its component parts, questions for key informants from the generic list were modified and tailored in accordance with what individual respondents could contribute regarding BALANCED. In addition, certain adjustments were made, particularly during site visit opportunities in Tanzania. For example, in lieu of face-to-face interviews with one or two key informants, at times the evaluator met with groups of project interlocutors and beneficiaries. While somewhat staged and stilted (translation was needed), the evaluator used these opportunities to discuss in more general terms life circumstances and potential changes within the purview of the BALANCED Project. In addition, observations made during several days of field visits, including observations of project staff in action, their interactions with one another, and the interface with implementing partners, provided additional dimensions to this evaluation. The four major components of the methodology used in this evaluation were as follows: 1. Document Review: The BALANCED Evaluation began with a review of a comprehensive set of BALANCED and related PHE documents. These documents, provided by USAID/PRH at the start of the assignment, included a number of internal project documents, publicly available reports, and selected websites. Supplementary documents also reviewed included selected published papers, program and other reports, unpublished technical papers, data compilations, presentations, and trip reports. During the key informant interviews, site visit to the Coastal Resource Center at the University of Rhode Island, and country visit to Tanzania, the evaluator collected and reviewed additional documentation produced by or pertinent to BALANCED, implementing partner organizations, and others working in the fields of population, health, and the environment. A list of seminal documents reviewed and most pertinent to this evaluation is included as Annex B of this report. 2. Interview Key Informants: A key informant is defined as someone with breadth or depth of knowledge about this project or the broader field of PHE. Most key informants were within USAID, the BALANCED Project itself, and partner organizations implementing family planning, health, or environment programs. In addition, some key informants largely unconnected to the project offered broad perspectives and PHE expertise. The evaluator conducted semi-structured interviews using open-ended questions to elicit relevant information from these key informants. Based on the SOW, background documents, and guidance provided by USAID/PRH, key questions were extracted from a list developed in relation to the seven SOW questions for each interview or set of interviewees. The list of key informants interviewed for this evaluation is found in Annex C of this report and the generic question list used for key informant interviews is provided in Annex E. Most key informants were suggested by USAID/PRH or the BALANCED Project. Additions to this initial list were made, particularly of higher-level officials within USAID and to reach out beyond the project to other experts in the field. Some key informants were interviewed more than once. To elicit information relevant to completing this SOW, the evaluator asked other questions either not on or modified from generic question list, as appropriate for particular key informants. In-person interviews of key informants were preferred. For this reason, a number of key informant interviews occurred during the week of June 13 to 17, both in Washington, D.C., and in Rhode Island, along with a number of other interviews in Tanzania. Remote phone or 4 MID-TERM EVALUATION OF THE BALANCED PROJECT email interviews were also used to access key informants in some cases, including those working outside of Washington, D.C., and Rhode Island, or those involved with PHE in countries other than Tanzania; in addition, several interviews and follow-up discussions occurred after the evaluator’s return from Tanzania. The evaluator’s initial meeting with three BALANCED staff on June 14 provided an opportunity for a PowerPoint presentation of a self-assessment (see Annex G) requested by USAID/PRH and conducted by BALANCED in advance of this evaluation. Annex H includes a table showing BALANCED Project indicators and progress toward targets. 3. Site Visits: The evaluator traveled to Tanzania from June 24 to July10 to visit BALANCED project implementation sites, interview key informants, and meet with partner organizations. Sites visited included communities within the Pangani and Bagamoyo districts where BALANCED currently has or plans to implement project activities. A site visit was also made to a potential partner organization, Longido Community Integrated Program (LOOCIP) in Longido District, Arusha, which implements related health and environmental activities. Key informants interviewed in Tanzania included USAID/Tanzania personnel in Dar es Salaam, in-country BALANCED staff, other international and local NGOs, and government counterparts. A schedule of the two-week visit to Tanzania is attached to this report as Annex D. Site visits to Bagamoyo, Pangani, and Arusha provided opportunities for the evaluator to collect a different level of data than that yielded by key informant interviews. The evaluator kept field notes on site visit observations and discussions. 4. Data Analysis: The evaluator concurrently recorded by written notes responses of key informants and subsequently transferred the notes to computer files. To analyze these, the evaluator comprehensively reviewed all key informant responses as well as conducted content searches for key terms and concepts. In addition, a review of site-visit field notes provided an additional layer of information. Based on this and as related to each question posed in the scope of work, trends and themes were identified. Preliminary summation for each SOW question was checked against information from project documents and other sources, including information learned during the site visits. Triangulation occurred among documents indicating what BALANCED was meant to do or accomplished, information reported by key informants on BALANCED, and information gleaned by the evaluator from the combination of document review, interviews, and site visits. A synthesis of responses to the four SOW questions that frame this part of the evaluation is found in Annex F of this report. MID-TERM EVALUATION OF THE BALANCED PROJECT 5 III. OVERVIEW OF MAJOR FINDINGS OF THE EVALUATION The implementing partners, CRC, CI, and PFPI, have faithfully and comprehensively implemented BALANCED toward the specified project objective: Advance and support wider use of effective PHE approaches. The project has organized around achieving three intermediate results, also specified in the RFA—IR 1: Capacity built for integrated PHE implementation; IR 2: PHE knowledge and tools developed, organized, synthesized, and shared; and IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity. PROGRESS TOWARD INTERMEDIATE RESULTS AND PROJECT OBJECTIVE: With direction and contributions from CRC, PFPI has maintained primary responsibility for capacity building (IR 1) and field projects (IR 3). Under knowledge management (IR 2), CRC in collaboration with the Johns Hopkins Center for Communications Programs, provides the developmental lead in organizing the website (http://www.k4health.org/toolkits/phe), with CI assuming some responsibilities for content on and communication about the website. Collectively, the BALANCED partner organizations have produced some progress within each intermediate result. However, as staff within each organization attest and performance monitoring plan (PMP) indicators show, project achievements have for the most part come more slowly than expected. Those indicators that are relatively easier to track and amass—for example, technical assistance provided or numbers trained—are on track with or have surpassed targets. (As one key informant noted, ―Who in Africa does not want to receive more training?‖) In contrast, other more significant indicators of progress, such as the number of field implementation projects receiving support and the amount of funds leveraged, lag behind expectations. More difficult to measure is progress toward achieving the project’s higher-level aspirations embodied within the project objective. The extent to which PHE itself, or USAID’s global leadership for PHE, has advanced as a result of BALANCED was not strongly supported by most key informants interviewed. Key informants outside the project who have been strong proponents of PHE expressed continuing hope and will for the success of BALANCED, as well as disappointment that it has not done better. A number of key informants, including those closest to the project, were perceptive about the weaknesses and shortcomings that had been observed. Significantly, the observations provided by such knowledgeable outside informants were often similar to the responses of key informants within the BALANCED Project itself. Though the depth of insight varied, both groups pointed to significant shortcomings within the project. By project’s end—less than two years from now—these and other significant shortfalls are likely to remain. Why this occurred became the principal focus of this evaluation and was traced back to certain structural weaknesses as well strategic shortcomings within the project, both as designed by USAID and as implemented by BALANCED. Before turning to the strategic dimensions of BALANCED, a description of the project’s structure is provided below. PROJECT DESIGN AND STRUCTURE: As designed by USAID/PRH and described in the RFA, the project was ambitious from its outset. In light of this fact, the amount of funding allocated was small. A number of key 6 MID-TERM EVALUATION OF THE BALANCED PROJECT informants, including several outside the project, made the same observation: BALANCED is over-designed and under-funded. Three years along, BALANCED remains ambitious and funds continue to be scarce. This situation is exacerbated by slow progress on receiving complementary buy-ins from within USAID and supplementary support from other donors. Clearly, these challenges were built in and evident in the RFA itself. CRC’s proposal mirrored the RFA, providing innovative approaches for implementing what was requested and bringing together partners with unique and complementary experiences for moving PHE forward. The proposal was understandably more optimistic than analytical about how built-in design challenges and difficulties would be faced and overcome. Within the competitive process, it is difficult for a bidder, especially one new to USAID/PRH, to take issue with weaknesses and shortcomings in the RFA. Ultimately, however, the project’s design and how BALANCED proposed to implement it, resulted in certain flaws that would prove difficult to overcome. A 2002 Congressional mandate directs that funding should be allocated for family planning within areas high in both population density and biodiversity. In line with this, USAID/PRH allocated $5 million of core support for this five-year project. The perspective of many key informants and the evaluator is that this amount was inadequate for what the project design envisioned. In addition, there were limitations on how these funds could be used: for field implementation, these central USAID/PRH funds could only be used to promote family planning. Family planning is only one component within PHE. USAID/PRH’s intention was clearly to encourage that funds from other sources be leveraged for field implementation. But this restriction on core funding also had the effect of constraining the PHE model subsequently implemented. Family planning would become the lead in fielding PHE, thus limiting the likelihood that PHE would be advanced as a comprehensive approach. The unique experiences and capabilities of PFPI as the field implementation partner within BALANCED further emphasized the focus on family planning. PFPI’s lengthy and extensive experience in implementing PHE at the field level brought to BALANCED an impressive level of expertise. However, as some key informants reflected, including within PFPI itself, PFPI’s approach is but one of several ways that PHE may be fielded. PFPI’s deep experience in the Philippines has concentrated almost entirely on family planning as a key intervention to mitigate the impact of high population density and growth on environmentally fragile areas. PFPI brought expertise in family planning to its pioneering work in population and environment, most notably in the IPOPCORM Project, which it complemented through work with local environmental organizations. Thus PFPI’s experience and expertise fit well within USAID/PRH funding restrictions—family planning would be the primary intervention introduced in BALANCED field project settings. PROJECT STRATEGY AND SHORTCOMINGS: In the RFA, the higher-level goal was ambitiously specified in the project objective: ―Advance and support wider use of effective PHE approaches.‖1 The three intermediate results (capacity building; knowledge management [K]; and field implementation) were then posited as the pathways toward achieving the project objective. From the evaluator’s perspective, neither the RFA nor CRC’s proposal sufficiently showed linkages among these IRs and connected these to the project objective; this contributed to strategic shortsightedness in subsequent 1 The seemingly small addition of the word ―worldwide‖ to the project objective in CRC’s proposal may have inadvertently led BALANCED toward grander ambitions centered around knowledge management than more realistic aims focusing on field implementation. MID-TERM EVALUATION OF THE BALANCED PROJECT 7 implementation. However, had CRC as the prime taken the lead in strategically planning and building a team, this would have gone a long way in overcoming this gap and preparing BALANCED for implementation. Unfortunately, such leadership was lacking. Some additional strategic flaws or weaknesses in the RFA should also be considered. For knowledge management, it is not clear that even well-executed KM will in fact lead to better use of available PHE guides and tools in the field. Furthermore, as experience to date within BALANCED demonstrates, such efforts are both time-intensive and resource-extensive. Reports that improvements had occurred in organizing and making PHE materials available on the website did not correlate with evidence that practitioners are actually accessing this resource. In addition, presuming that Missions would be willing to build on seed grants to scale up PHE may prove wishful thinking. For one thing, with field implementation resources limited to some $20,000 per year, it is difficult to field a PHE developmental model of sufficient scope to attract interest. Clearly it would not be possible to do so without extensive effort to know various Missions and understand their strategic priorities. Missions could well have interest in using PHE to bring FP to remote, underserved regions, but not much incentive to test the PHE model or its scale up. Furthermore, the integrated approach has always been confounded and perhaps will always be blocked by stove-piped funding streams. With these RFA considerations as background, this overview now shifts to how CRC and the partner organizations proposed and then organized to implement BALANCED. In the proposal itself, CRC put forth a vision for BALANCED: “Cadres of competent PHE champions and practitioners from Africa and Asia are PROMOTING the comparative advantage of approaches that simultaneously support family planning and conservation, are working collaboratively with other groups GLOBALLY to apply PHE knowledge and state-of-the-art practices in biodiversity-rich areas, and actively documenting and sharing evidence of added-value and better practices of PHE using 21st century knowledge management and learning tools.” Unfortunately, after the project was awarded, this vision—an important, albeit ambitious, underpinning for strategic planning—fell by the wayside. As reported by several key informants, those most involved in writing the BALANCED proposal ended up least involved in planning its implementation. Such separation between proposal architects and subsequent project planners is not unusual. Furthermore, this occurrence does not necessarily or often result in projects devoid of strategic planning or direction. However, as widely reported by key informants both within and outside of BALANCED, strategic planning and direction have been key missing ingredients in the project’s subsequent implementation. Critical for implementing any project is to organize at the outset available resources, financial as well as human. This facilitates aligning component parts toward achieving the higher-level goal. Establishing a unifying vision, setting priorities, and making choices among the many things that could be done are essential elements within strategic planning. What emerged during this evaluation from interviews of key informants, most notably from many within the project, is that BALANCED even now lacks an overall vision and strategic plan. Though both vision and strategy development received some attention toward the end of the first year, BALANCED has been more task-oriented than strategic from the outset. The RFA, which called for the First Year Work Plan to be completed within one month of the award, was complicit in encouraging immediate attention to detail. Key informants reported and internal documents confirm that BALANCED immediately plunged into developing its initial work plan and soon thereafter a PMP. In the process, several essential steps were missed: establishment of a shared, unifying vision; a conceptual framework 8 MID-TERM EVALUATION OF THE BALANCED PROJECT interlinking IRs, sub-IRs, and activities within the capacities of partner organizations; a strategic plan to guide resource allocation and implementation timing; and cross-cutting through and around all of these, team-building to move the project forward. Such steps were of particularly critical importance to the future success of BALANCED, a project that brought together organizations based in three different locations that had not previously worked together. A strategic plan for BALANCED could have usefully considered the project’s future implementation using three key questions: Who should BALANCED serve? How are we as a team to excel in its implementation? Especially considering funding limitations, what activities should we prioritize within the project? Facing a complicated project with limited resources, project leadership immediately got to work, but unfortunately there was no roadmap. The First Year Work Plan, completed in January 2009, set the standard for those that followed. This lengthy (57 page), detailed compendium of activities is devoid of strategic explication, lacking an overview or rationale for why the specified amounts of resources were being used at that point in the project for the outlined activities. Clearly, the BALANCED Project team made some attempt to think strategically about PHE and the project. Late in its first year, diagrams were developed to prompt strategic thinking within and around BALANCED. However, these did not end up being used to strategically guide BALANCED inputs toward outputs and outcomes.2 An explicit strategy for implementing BALANCED was never embraced; if an implicit strategy has guided the project, it is not apparent to most staff. The Advisory Committee Meeting held toward the end of the first year was one occasion where the project’s dimensions were analyzed in strategic terms. This two-day meeting at CRC brought a high-caliber group of outside experts together with core BALANCED staff. By all accounts, this was an engaging event. Many key informants who participated, inside as well as outside of BALANCED, cite this event as a high point for the project. Notes from the meeting show that participants provided a number of useful, strategic suggestions for the project’s consideration. However, in organizing the meeting, the BALANCED team decided to focus on IR 2, Knowledge Management. Though advisors offered inputs across a wider range of areas, only those within KM seemed to be taken into consideration as the project moved forward. What was lost in the process were a number of interesting insights and inputs that could have more broadly assisted and aided the project. For reasons not clear (though budgetary constraint was the most frequently cited rationale) and despite intentions to tap further into this rich resource of willing advisors, this turned out to be the only Advisory Committee meeting that BALANCED convened. PROJECT IMPLEMENTATION AND IMPROVEMENT: BALANCED faced communication challenges from the outset. The three partner organizations were both new to and distant from each other, with one located half a world away from the others. As prime, CRC of the University of Rhode Island was new to USAID/PRH and, like the new director it hired, also new to PHE. Furthermore, the project director operates out of a one-person office in Washington, D.C., far from the Rhode Island base of CRC. After a slow start, BALANCED has worked hard to improve communications and team cohesion within an admittedly difficult structure, with even vociferous critics within the project reporting that improvements have been made. However, three years into the project, the situation remains 2 A more recent indication of strategic thinking within BALANCED may be found in the conceptual model on linkages among PHE components and livelihood development provided in the latest BALANCED newsletter. MID-TERM EVALUATION OF THE BALANCED PROJECT 9 that BALANCED does not have a strategic roadmap to guide how, which, and when component parts contribute to its higher-level goal. Project staff at all levels expressed frustration with how decisions are made about what is to be done, and when and where. Furthermore, though there was and is much communication about project implementation, confusion sometimes reigns. Lines of responsibility end up being blurred. Management of the details has led to over-communication on issues of lesser importance while the larger picture remains incomplete. To cite one example, from the beginning it was clear that, to be successful, BALANCED would need to leverage additional funding to support field implementation and scale up. In addition, the RFA provided guidance on priorities when it stated that ―knowledge, tools, and understanding will be generated by these field activities as well as local leadership and champions developed among the organizations implementing PHE activities.‖ (RFA, p. 32). Within BALANCED, however, initial priorities were set the other way around—with early concentration on KM and training programs to develop capacities, less focus on identifying seed grant opportunities, and little attention to identifying and attracting additional resources. In retrospect, clearly USAID Missions and other funding sources should have been a high priority from the project’s beginning. Determining the interests and priorities of complementary donors, and assessing the potential for attracting additional funds, should have been featured activities during the first year. Instead, though some meetings with Missions did occur, approaches were made without sufficient understanding or consideration of the Mission’s own priorities, as recently transpired in Tanzania. Though several other donors were also approached, lack of success may similarly mean that BALANCED was not operating on the same strategic plane. Upstream work with potential funding streams could have positioned BALANCED within a leadership role for PHE. Lacking such effort early on, BALANCED was not included in strategic discussions toward forming an alliance of USAID with the Packard and MacArthur Foundations on collaborative contributions to support PHE in Africa. These shortcomings—which are those of CRC as the prime for BALANCED—started at the beginning of the project and have persisted to date. Is it too late to rectify? The recent call for seed grant proposals, which resulted in some 15 submissions, was a belated move in the right direction. However, at this late stage with no supplementary funding sources lined up, this may well lead to frustration about PHE rather than an increase in proponents and contributors. Even the presentation of the project’s self-assessment was similarly heavy and immediate on the details of implementation during the initial contact between the evaluator and the BALANCED core team of director and two deputies. No overarching strategy for BALANCED was evident or articulated. Though candid about specific shortfalls, the self-assessment made no attempt to place the myriad of project activities within a broader context. As is apparent in the self-assessment, there is need for BALANCED to catch up. In subsequent sections of this report, suggestions are offered on steps that have been and still may be taken to bring this project in closer alignment with its higher-level aspirations and goals. First, however, the following section reviews and reflects upon activities and progress within the project that are areas of strength and accomplishment. 10 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 11 IV. POSITIVE PROGRESS AND REMARKABLE RESULTS In its first three years, BALANCED has made significant progress in several areas and within some indicators in relation to all three project IRs. In compliance with the project objective, these accomplishments are contributing to advancing PHE. Some notable achievements are highlighted below. RESPONDING TO FIELD PROJECT OPPORTUNITIES: Though fewer than anticipated, BALANCED has been able to develop and support interesting field experiments in PHE. In Ethiopia, Ghana, Tanzania, and Zambia, variants of the PHE model are being tested in decidedly different contexts. CRC opened the way to including PHE within its ongoing environmental programs in Tanzania and Ghana. Though this evaluation only gathered in-depth information for field implementation in Tanzania, key informants knowledgeable about and working within Ghana, Ethiopia, Zambia, and the Philippines provided perspectives on BALANCED activities in other countries. Contacts made at the country level and with environmental organizations led to the award of seed grants for PHE implementation in Ethiopia and Zambia. As a result, PHE is currently being applied in forested as well as farming areas, for fisheries as well as for coastal management. BALANCED implementation in Ghana and Tanzania builds on nutrition interventions and HIV/AIDS prevention supported by Feed the Future and PEPFAR funds, respectively; in Zambia BALANCED provided assistance on a successful proposal for Flex Funds. Furthermore, BALANCED is currently reviewing proposals for additional field projects and is likely to support two or three more in the project’s remaining time. In addition, the buy-in received from USAID/Philippines has enabled BALANCED to scale up its already proven PHE model to new areas of the Philippines. Taken together, field implementation under BALANCED is adding to our experiential knowledge for applying PHE in different settings. However, it remains to be seen whether PHE will be an interesting add-on to existing programs or fully embraced for the added value it potentially provides. KNOWLEDGE MANAGEMENT—UPGRADING THE PHE KNOWLEDGE BASE AND TOOLKIT: BALANCED successfully reorganized the existing PHE knowledge base and made it available on the K4Health website (http://www.k4health.org/toolkits/phe). In collaboration with Johns Hopkins CCP, BALANCED staff from CRC added a level of expertise that significantly advanced what was previously available on the Internet and searchable on PHE. In one location, many of the most practical, applicable, and usable findings from PHE programs worldwide may be readily found. Consulting this website, one can quickly get a primer of the most important information available. The site is in addition fairly easy to access and navigate. While purposively oriented toward information of practical use, one misses more conceptual or theoretical underpinnings of the approach. Furthermore, though Hopkins reports steady increases in accessing the site, the question of how much this resource is actually used by practitioners for field implementation of PHE remains unanswered. Most key informants questioned during this evaluation about the use of this website reported that either they were sufficiently aware of the materials available as to not have need for the site or that they simply did not access the website. The latter was the response of almost all in Tanzania who were interviewed and may represent those in other countries where BALANCED is being implemented. Aware that the site existed, often as a result of participating in a 12 MID-TERM EVALUATION OF THE BALANCED PROJECT BALANCED training program, the website was still not accessed. Informants often cited connectivity as well as electricity as important determinants of use. Distribution of pen/thumb drives to compensate for such obstacles did not lead to higher use; BALANCED reports that a number of pen/thumb drives malfunctioned and the one or two key informants reporting that they had received such drives had yet to use them. A useful end-of-project summary by BALANCED would be to revisit use and users in light of a more fully populated website and make recommendations about target audiences and how to reach them in the future. INTEGRATING FP AS A PHE INTERVENTION: With USAID/PRH seed grant funding largely restricted to implementing family planning, BALANCED has done well to find opportunities for layering FP on top of other interventions already in the field. This has in fact offered some unique advantages, such as observing how difficult or easy it is to insert family planning into ongoing projects. In Tanzania, for example, BALANCED leveraged ongoing work with micro-credit associations for alternative livelihoods to benefit FP promotion and use. Many community-based distributors (CBDs) and peer educators (PEs) in the Pangani implementation sites as well as those soon to be trained as PEs for expansion sites in Bagamoyo (and perhaps even some youth peer educators [YPEs]) are active in these savings and credit cooperatives (SACCOs). As proponents for family planning who provide or link to contraceptives, they are well positioned to mobilize community members to access these needed services. CAPACITY DEVELOPMENT FOR FIELD IMPLEMENTATION OF PHE: Inviting potential partners and collaborators to trainings on various aspects of PHE design and implementation has helped develop individual and organizational interest in PHE. All those interviewed in Tanzania who had participated in PHE trainings of various kinds (information, education and communication [IEC], behavior change communications [BCC], CBD training of trainers [TOT], PE TOT, PHE design) attested to the high quality of training received and expressed appreciation for the opportunity. A reported—though perhaps inevitable— shortcoming was that not always the ―right‖ persons participated or that, once trained, individuals did not remain long in the positions held; moreover, unless directly participating in a PHE project, participants rarely reported actually putting what they had learned into practice. Nevertheless, inviting participants outside of projects and from other countries, as has occurred with regularity in Tanzania trainings, has served to spread interest in and develop capacity for PHE. Interestingly, the most significant achievement of BALANCED for capacity building may be what has occurred with PFPI itself as it adapted its skills and experiences to serve the very different contexts of Africa. MAKING FAMILY PLANNING INTEGRAL TO CONSERVATION: By participating in BALANCED training programs and workshops, some environmental organizations have effectively engaged with population and FP as part of the broader development context. Similarly, technical assistance and support for field projects has also directly involved some conservation programs in providing FP as part of their activities. Organizations interviewed as part of this evaluation that reported being positively affected by BALANCED include the Africa Wildlife Fund, the Tanzania Coastal Management Program, the Wildlife Conservation Society, and the USAID-funded Integrated Coastal Resource Management Initiative for Western Ghana. In addition, a participating Pathfinder representative expressed appreciation that PFPI joined a planning workshop to explore collaboration with The Nature Conservancy and the Frankfurt Zoological Society on PHE. MID-TERM EVALUATION OF THE BALANCED PROJECT 13 Within field projects that it directly supports, BALANCED efforts in FP are more layered than integrated. However, adding contraceptive information and services to ongoing environmental projects is a viable approach for reaching prospective clients in remote and environmentally fragile areas. This aspect of the project accords well with the Congressional mandate to extend family planning to biologically diverse settings and with USAID/PRH’s efforts to expand access to FP. STUDY TOURS: The Philippines study tour conducted early in the project was pivotal to spreading practical knowledge about PHE. Organized by CRC and PFPI to spread interest among Africa-based counterparts on the possibilities and potential for PHE, participation in this event was cited by many key informants as a high point for the project. Though USAID/PRH initially resisted this use of project funds, it came to realize and appreciate the value of developing first-hand contact with what has been called a ―gold standard‖ project, IPOPCORM. Though expensive, when well planned and properly executed, such South-South exchanges appear worthwhile. The Philippines study tour was another important point of departure for BALANCED to become invested in developing PHE capacities among Africa-based counterparts. Intra- and inter-country study tours may now afford similar opportunities. A group from Ghana recently visited TCMP to learn about coastal conservation and micro-credit. While not sponsored by BALANCED, the positive outcomes from this exchange encourage other similar efforts. For example, TCMP is being thought of as a useful site to accelerate start up of other seed projects in the country and the region. SOUTH-SOUTH COLLABORATION AND ASSISTANCE—THE ROLE OF PFPI: A compelling part of the proposal was that BALANCED would tap into PFPI, one of the most experienced organizations in implementing PHE. This use of PFPI as the implementing partner for field projects, and to a great extent capacity development, is a project strength. By effectively engaging PFPI, a Philippines-based organization, with PHE capacity building and field implementation in Africa, BALANCED embodies one of the best on-the-ground examples of South-South collaboration and assistance. Candid appraisals from PFPI itself, as well as from other knowledgeable informants, clearly indicate that PFPI faced a steep learning curve in adapting its experiences to Africa’s different and varied context. PFPI’s PHE model, including extensive work with community-based volunteers and local governments, was not readily transferable. Furthermore, unlike many conservation organizations and some other PHE practitioners, PFPI does not go deep from the outset to involve communities in shaping PHE and its interventions. Even so, over time and to an extent, PFPI has adapted its approach, with field projects currently being implemented in four countries and technical assistance for PHE extending to an additional two countries. CRC staff members have ably contributed to these efforts. Africa-based key informants interviewed for this evaluation were unanimous in praise for the expertise that CRC and principally PFPI brought to PHE project design, training, and technical assistance. 14 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 15 V. MISSED OPPORTUNITIES Hindsight is a unique and privileged perspective of the evaluator. Nevertheless, it offers an opportunity to surface issues and omissions that may be adjusted as the project continues, or addressed in the re-design of a subsequent iteration. In this section a number of opportunities adjudged to have been missed up to this point in project implementation are listed and described. These surfaced either directly from key informant interviews or during analysis. Remedial action or long-term effort to redress these would likely improve this project and enhance its implementation. Strategic Planning—A number of key informants within the project stated that BALANCED lacks a clear, unifying vision of what it aims to accomplish and a strategic plan for getting there. Responding to general questions about strengths and shortcomings within BALANCED, several key informants cited such strategic-level weaknesses and omissions. Some selected quotes from those within the project follow: ―Key to any project is a strategy meeting that gives a vision and everyone knows what to do…(this) should have been key in the beginning.‖ ―We operate on a yearly basis and have not been strategic.‖ ―A systematic game plan is not there. How are we going to promote PHE worldwide? We are doing it by bits and pieces, lacking strategic thinking.‖ Key informants outside of BALANCED made similar observations: ―(The Project is) scattered—not strategic.‖ ―BALANCED does not have systems thinking…(It) seems to approach things piecemeal…Not impact-oriented.‖ ―BALANCED has lacked strategic thinking early on about what donors and Missions are most interested in.‖ In the BALANCED proposal, CRC did put forth a unifying vision and a basic framework that aligned the project objective, IRs, and sub-IRs. Even in its acronym, Building Actors and Leaders for Advancing Community Excellence in Development, BALANCED set a potential guidepost for high-level aspirations. Once awarded, however, BALANCED neither revisited the strategic elements within its own the proposal nor formulated a comprehensive strategy to guide implementation. Rather, taking its lead from the RFA’s stipulation for a first year work plan within 30 days of award, BALANCED busied itself with the details. Could USAID itself, either in the RFA or subsequently, have been more assertive about the need for a strategy to guide implementation? USAID/PRH made concerted efforts in this direction toward the end of the first year of BALANCED. Encouragement for BALANCED to become more visionary and strategic produced some short-term gains in the form of a vision emblem and some strategic diagrams. However, as evinced in the recent responses of many key informants, these have not had lasting impact. Experienced leadership within BALANCED should have recognized and reacted to the need for a common vision and a strategy to guide implementation. Concentrated, early focus within the project to strategize across and among the IRs would have led to efficiencies and progress toward realizing the project objective. Clearly, this would have set some priorities and provided a better basis for negotiation with USAID on the allocation of scarce resources. 16 MID-TERM EVALUATION OF THE BALANCED PROJECT Team Building—Lack of strategic planning was mirrored by shortcomings in building the team that was to implement the project. Especially due to distance among implementing partners— which involved spatial separation as well as lack of joint work experience—dedicated effort to build at least the core team was essential. Project start-up provided an opportunity to help staff understand how the component parts—the three partners as well as the three IRs—would intersect and operate to collectively contribute to achieve the higher-level objective. What happened instead was work plan development. Though this involved some collaboration among the partners, the focus was already into the details, the weeds of implementation. The opportunity was missed for the team to deliberate and coalesce around how specific activities related to higher-level goals and aspirations. Some belated opportunities for team building were also missed. Core team participation in the Philippines study tour on the one hand, or greater representation of staff working at field level in the Advisory Meeting on the other, could have both served this purpose. At the time of these events, a convincing argument for USAID/PRH to approve the added travel costs could have been made in terms of advancing strategic thinking within BALANCED. As it turned out, however, the trees and seascape, and beyond that the forest and ocean, were not in view. Timely and Strategic Support for PHE Field Projects—A call for proposals for PHE seed grants was not issued until well within the third year of BALANCED. For a project of five years’ duration, waiting until well past the midpoint was neither timely nor strategic. If learning from seed grants and other support provided to PHE field projects is to filter into the knowledge base, a concerted effort to identify opportunities should have started far earlier. In addition, early respondents to such proposal calls could have formed the pool of organizations interested in capacity development. Furthermore, strategic decision-making does not seem to have informed where and to whom seed grants would be made. BALANCED has clearly been opportunistic rather than strategic in identifying PHE seed grant opportunities and partners. This is not to say that ongoing support for CRC-affiliated programs in Tanzania and Ghana will not prove useful. However, during the first years of BALANCED, it was as if individuals and organizations that had met in passing largely became the pool within which seed grant applications were encouraged. Leveraged Funding—Given project funding constraints and USAID encouragement to seek supplementary funding for field implementation, clearly more time early in the project should have been given to identifying potential funding sources, particularly though not exclusively within USAID Missions. Understanding donor priorities is fundamental to making successful appeals for funds. Even in Tanzania, where BALANCED has had its most extensive experience in Africa, the recent failure to attract Mission funds to ―scale up PHE‖ was fundamentally due to making the case in terms of BALANCED rather than Mission priorities. Nor has the project tapped into other funding sources. An exception is the buy-in to BALANCED from USAID/ANE for activities in connection with climate change, which BALANCED staff dismissed as a pass￾through in which they were not actively engaged. BALANCED has piggybacked on funding already in place, such as USAID/Tanzania support for the Pwani Project in Pangani to continue TCMP environmental efforts and extend these for HIV/AIDS prevention, and USAID/Ghana support to CRC from Feed the Future to include a nutritional component within its ongoing coastal fisheries program. However, BALANCED has not attracted funding on its own. Failure early on to identify and secure funding has affected what the project can ultimately accomplish. This shortfall hampers efforts to replicate PHE in new sites, scale up existing field projects, and have greater presence in the field. Correspondingly, indicators tracking the number of field implementation projects receiving support and the amount of funds leveraged are behind expected targets. MID-TERM EVALUATION OF THE BALANCED PROJECT 17 Advisory Committee and Advancing High-level Leadership for PHE—BALANCED received accolades internally and externally for convening a high-level committee of PHE advisors one year into implementation. By all accounts, the meeting at CRC’s campus at the University of Rhode Island was high powered and highly engaging. Though focused on IR 2, Knowledge Management, organizers and participants alike were open about providing input on a wider range of issues pertaining to PHE and project implementation. An immediate output of the meeting was that BALANCED received a range of useful advice. In the process, BALANCED also gained traction and credibility among an influential group of PHE leaders and advocates, a few of whom were erstwhile competitors. However, BALANCED failed to build on the momentum and good will the meeting had generated. Despite indications otherwise, no subsequent meeting of advisors was held. Though occasionally contacted for input on specific aspects of the project, advisors interviewed for this evaluation expressed a resulting sense of alienation from the project. This missed opportunity, which BALANCED attributes to lack of funding, has harmed the project among a powerful group who were at one point willing and open to serving as strong proponents. Health, the Missing PHE Component—To date, despite encouragement to do so within the RFA, BALANCED has leveraged no funding from USAID-funded health projects nor has it inserted the PHE approach into ongoing health programs. This contrasts with considerable efforts made to apply PHE within ongoing environmental programs. Yet there is significant need for health interventions at the community level, as well as a clear need for funding for various types of interventions, from health systems to water to nutrition. In Tanzania, Mission funding for the Pwani Project extends TCMP’s environmental efforts to include HIV/AIDS prevention and perhaps give some attention to water and sanitation. However, BALANCED activities in Pangani and Bagamoyo serve more to recognize that such parallel efforts are taking place than to actually build an integrated approach upon such efforts. The omission of health from PHE is critical, not just within BALANCED but also more broadly for advancing PHE. In part, this can be traced to the PE origins of many organizations, including PFPI, now involved in PHE. Perhaps derived from this, as the evaluator observed in Tanzania, FP is not sufficiently emphasized as an intervention that improves the health of mothers as well as children. At the same time, PFPI and BALANCED staff seemed to give priority in trainings to population and environment, with little attention to health. Attention to population issues and the reduction of growth rates is an important, albeit more macro-level, dimension of PHE. At the community, family, and individual level, particularly in Africa, family planning is more salient and readily accepted as an important, immediate health intervention. Building the PHE Knowledge Base—In addition to generating field-based knowledge about PHE, the university base of BALANCED offered potential for deeper research and understanding of PHE. Unfortunately, an initial foray of university staff to conduct value-added research on PHE in the Philippines created more rifts within the project, and more broadly within PHE as a field, than it did interest in learning more. Most of those interviewed said that, though somewhat burned by the experience, they had learned from it. Conclusions that emerge from this outside look at PHE are that more community participation, greater transparency, and more communication between researchers and implementers are important ingredients for advancing PHE. After some delay and much discussion, findings from that study are being finalized for publication. Hopefully, the university as a resource may still be tapped to increase understanding of PHE and USAID will support this. Conceptual work on interrelations between livelihoods and PHE, which appeared in a recent newsletter, is a step in this direction. Earlier and more open communication between university researchers and PHE implementers may help in future undertakings. 18 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 19 VI. CATCH-UP POSSIBILITIES BALANCED will end in September 2013. The Year 4 Work Plan (July 1, 2011, to June 30, 2012) has been developed and approved. Nevertheless, there is time and room for some end-of￾project adjustments. One highly placed key informant inside the project suggested ―a big hoopla and hurrah‖ to end the project in Year 5. However, rather than the usual big-splash meeting to trumpet project accomplishments, which are usually pro forma and often fall flat, the evaluator suggests that USAID consider the following suggestions to build momentum for PHE BALANCED: Strategic Planning—Both USAID and BALANCED need to move above the weeds of IR activity implementation to a view of the larger forest of PHE advancement. It is possible that a strategic summit to frame and guide implementation during the time remaining in the project would be useful; such a summit would involve selected field staff, advisors, and project personnel in an intensive effort to identify high-priority targets for the project’s last year. This may also provide opportunity to bring BALANCED advisors back into the picture. In addition to some showcasing for BALANCED accomplishments, this could also serve as an honest appraisal of challenges and shortcomings. The purpose of the meeting would be to prioritize those activities and investments that would strategically advance PHE in the coming year, if not in coming years. Management Review—A useful outcome of this evaluation would be for BALANCED to adjust staff roles and responsibilities to maximize the potential for achievement within the time remaining in the project. This does not mean that staff should simply be encouraged to do more, faster. Rather, BALANCED should adjust its management configuration to eliminate redundancies and inefficiencies, clarify implementation and reporting responsibilities, and solidify collaboration across the three implementing partners. Staff were for the most part very candid with the evaluator about management and what is working and what is not: implementation has not had a strategic vision or plan; there is too much ongoing communication about line-of-work responsibility already clearly delegated; project details (such as editing) receive more attention than big picture items; full delegation and trust are sometimes missing. Some lines of responsibility need to be shifted, clarified, and consolidated. Some questions that may help move this management review forward are as follows: Should the CRC International Program Director have a greater role in BALANCED? Has having two deputy directors been useful and constructive? Does the project director need to shift more to the role of chief executive officer, who sets a vision and oversees a strategy? Should a deputy director be delegated to take over daily operations, more like a chief operating officer? Should CI have a different role within BALANCED other than what it does with IR 2? How can the lead technical director based in the Philippines more closely coordinate with CRC field implementation staff based in Rhode Island? How can the BALANCED East Africa consultant, who has the most understanding of BALANCED implementation in that region, be more involved in decision￾making (e.g., seed grant decisions) as it affects that region? Can work plans and semi-annual reports be streamlined for the remainder of the project? Develop Leaders and Leadership for PHE—BALANCED has made some modest efforts to support champions and leaders for PHE. Stories on its website about selected inspirational leaders and sponsoring PHE champions to present at international conferences (albeit only family planning conferences to date) are the major activities undertaken. Admittedly resources are scarce. However, CRC in particular is well positioned to do more. An annual course for PHE leadership and leaders would advance PHE among those most prepared and best 20 MID-TERM EVALUATION OF THE BALANCED PROJECT positioned to move the field forward. Such a program, developed and launched in conjunction with the last year of BALANCED, could be done at minimal cost to the project. CRC has the experience as well as the infrastructure to undertake a program of this kind. Its revenue￾generating certification course, Marine Protected Areas, offered annually, provides a good model. A program focused on PHE leadership development could draw on lessons learned within BALANCED as well as state of the art for PHE more broadly. To prepare a next generation of PHE leaders, considerable attention should also be given to developing individual and collaborative skills in leadership. Such a program, perhaps conducted in conjunction with PFPI and/or the more broadly based Environmental Leadership Program at UC Berkeley, could leave a lasting legacy for PHE. Conservation International’s Involvement—Conservation International brought implementation experience to BALANCED field-level PHE in several settings. Different from PFPI, which is an organization largely oriented toward family planning, environmental concerns are paramount for CI. It is precisely among similarly committed conservation organizations that PHE hopes to make headway; getting population and health mainstreamed into environmental programs. Unfortunately, within BALANCED CI appears to have been largely peripheral to date. Rather than tapping into CI’s strength area for integrated approaches, the project has relegated CI to a relatively minor role within the context of KM. Renewed interest for CI field activities in the Philippines as well as the recent opening of Madagascar could reverse this. It is important for BALANCED to facilitate CI’s having a broader role within the project. Advance Community Participation and Planning—Key informants both within and outside PFPI recognize that IPOPCORM is only one of several PHE models. The BALANCED approach, led by PFPI, remains largely one where communities are assessed for entry points more than for community involvement. Clearly, conservation groups understand and are more committed to deeper community participation than are organizations involved in family planning. As conservation organizations, such as the Jane Goodall Institute and World Wildlife Federation, recognized the importance of surrounding communities for preserving natural resource areas, community participation to plan and manage local environments became the accepted approach. CRC Ghana, for example, uses maps (aerial as well as locally generated) to involve communities in helping them understand and plan for environmental and other changes. Family planning organizations, in contrast, come with both curative medicine and preventive services as backgrounds. On the one hand, making contraceptives more widely available is always their approach. On the other, involving communities and particularly women in how services are provided will often lead to greater efficiencies and more choices. To more fully advance PHE, BALANCED should seek opportunities in Ghana and elsewhere for deeper approaches within PHE, approaches that are more community-intensive and intervention–extensive than it has undertaken to date. Extend Family Planning in Tanzania—The model now in place in Pangani, Tanzania, provides a solid basis for extending FP into other areas covered by TCMP. Plans are already developed to do this in additional wards and villages in nearby Bagamoyo. An important learning from implementation to date is that peer educators, rather than community-based distributors, are the keys to success. However, local BALANCED staff must be much more closely connected to others working in FP in Tanzania. Mobile teams combined with community mobilization have become important USAID strategies for ―catch up‖ in family planning. Participation in the monthly meetings convened by the Ministry of Health to coordinate among partners working for FP is essential. MID-TERM EVALUATION OF THE BALANCED PROJECT 21 Also key is a broader vision of what can be accomplished in the project’s remaining time. For example, instead of replicating in a few additional communities, the evaluator suggests a broader effort to expand access to FP across all areas where TCMP is operating, most notably including Zanzibar and Pemba. Mission priority on expanded access to a wide range of contraceptive services, combined with BALANCED’s ability to mobilize communities, could well result in much larger impact as well as leveraged funds from USAID/Tanzania. New Seed Grants—The number of organizations that responded to the project’s recent call for proposals is in itself an indication of continuing broad interest in PHE. It will be interesting to see what criteria BALANCED applies to select a small number from among the many applications received. From what the evaluator learned, interesting possibilities clearly exist. To improve implementation efficiency, perhaps BALANCED would select and support seed grants to work in tandem with joint workshops and training programs as well as exchange visits to learn from each other. Based on two seed grant possibilities that the evaluator learned about— in Mahale and Longido, Tanzania—this could prove a viable and useful approach. It would also contribute to building the network of practitioners that BALANCED is committed to. 22 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 23 VII. CONCLUSIONS In response to a Congressional mandate that family planning be promoted and provided within high-density, biologically diverse areas, USAID/PRH has been investing in PHE for nearly 10 years. With relatively small amounts of funding, PHE has developed as an interesting approach to serve remote, sometimes largely unreachable areas. As such, and even though PHE may not contribute large numbers of FP users, it has received consistent and considered technical support and oversight from USAID/PRH. The evaluation of BALANCED is the latest example that such due diligence is being given to PHE. The evaluation has focused on how and with what effectiveness this particular project has been designed and is being implemented. A previous USAID-funded project to advance website development and Mission-level interest in PHE did not achieve expected results. Had this prior project been more effective, the scope of BALANCED could have been more limited and progress somewhat easier to achieve. The ambitious but under-funded project design outlined in the RFA clearly created some obstacles for BALANCED to overcome. However, as this evaluation has documented, a number of shortcomings and inefficiencies within BALANCED were of its own making. Principal among areas where BALANCED has fallen short was failure to take a strategic approach to planning and prioritizing project implementation. Considerable strategic thinking and planning was needed at the outset. Except for some elements that appeared in the BALANCED proposal, this did not happen. The repercussions have been significant. Across a team as geographically and culturally diverse as BALANCED, lack of an overarching direction and vision has resulted in considerable levels of frustration, division, and isolation among team members. Though teamwork is evident within the larger implementing partners, CRC and PFPI, an overarching team that serves BALANCED is not in evidence. Many key informants from within BALANCED were openly critical about internal operations. That they were forthcoming in this way to an external evaluator strongly indicates the need for improvement. This is not to say that there have not been positive achievements and accomplishments. In addition to developing and populating a website that is highly regarded, if not yet as widely used as hoped, BALANCED has faithfully made some progress within each IR and on most indicators. It has significantly advanced a number of important development dimensions such as South-South collaboration, FP partnerships with environmental programs, and development of capacities with a number of new settings for PHE implementation. The evaluator has attempted to identify these dimensions throughout the report as well as to give a number of them specific attention in Section IV. However, the higher hanging fruit of providing global leadership for PHE or innovation in adapting PHE to new settings has not yet occurred. This level of field-building achievement is not likely to occur without significant adjustments within BALANCED in the time remaining or design changes by USAID/PRH for future undertakings. Some basic adjustments are needed, including establishment of clear priorities about what can significantly contribute to progress within BALANCED in the remaining years. In this report, particularly in Sections V and VI, suggestions emerging from this evaluation are provided as guidelines for project improvement. In a separate memo, the evaluator will offer USAID/PRH ideas on how it may want to change or restructure future investments of this kind. For BALANCED itself, however, the promising partners brought together for this project need to develop a clearer, shared understanding of what they are attempting to accomplish collectively and support to help them get there. 24 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 25 ANNEX A. SCOPE OF WORK BACKGROUND: As part of its efforts to support and expand Population, Health, Environment (PHE) programs worldwide, USAID’s Bureau for Global Health’s Office of Population and Reproductive Health (USAID/GH/PRH) awarded the Population, Health and Environment technical leadership cooperative agreement (GPO-A-00-08-00002), Building Actors and Leaders for Advancing Community Excellence in Development (BALANCED) to the Coastal Resources Center (CRC) at the University of Rhode Island (URI) on September 17, 2008. The BALANCED Project is supported by a highly qualified team of international partners, including PATH Foundation Philippines, Inc. (PFPI) and Conservation International (CI). The five-year BALANCED Project has a ceiling of $5 million in core funding from the Office of Population and Reproductive Health (PRH) in USAID’s Global Health Bureau and a potential additional $2.5 million budget for Mission buy-ins from all accounts. The AOTR for this agreement is Gloria Coe, the Technical Advisor is Heather D’Agnes, both at USAID, and the Project Director is Linda Bruce at CRC. The project started in October 2008 and will finish in September 2013. The Project’s overall objective is to advance and support wider use of effective PHE approaches worldwide. PRH envisioned the BALANCED Project as the flagship PHE project which would build on lessons learned and best practices from the previous 6 years of USAID/PRH investment in PHE (as well as lessons learned from other PRH and environment/natural resources management programs) in order to make major progress towards advancing and supporting the expansion of effective PHE approaches globally. The BALANCED project is managed by USAID/Washington with activities focused on the Coral Triangle region and on East Africa, and field presence based in the Philippines and Tanzania. The expectation is that at the end of the BALANCED project effective PHE approaches are documented, fine-tuned and readily available; an increased number of health and environment organizations are adopting and implementing PHE approaches; and a cadre of experienced PHE professionals and champions are available to provide technical support and advance PHE efforts. To accomplish this objective, BALANCED has three Intermediate Results (IRs): IR 1: Capacity built for integrated PHE implementation. BALANCED will provide technical assistance and build the capacity of implementing organizations to design, implement, evaluate, and learn from successful PHE projects with an emphasis on tapping south to south learning. The result of this activity is to develop a cadre of PHE champions who, through their successful implementation of PHE projects, can speak knowledgeably about how to implement PHE programs, the benefits of PHE as an approach, and can explain to others why and how it can be implemented. IR 2: PHE knowledge and tools developed, organized, synthesized and shared. BALANCED will gather, develop, and synthesize existing and new PHE knowledge and state of the art approaches and communicate that knowledge to other audiences. In addition to this, BALANCED will have a learning component to identify, analyze and document promising PHE practices and approaches, the added value of the PHE approach, and successful techniques for scaling up PHE approaches. IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity. BALANCED will support the delivery of small-scale integrated PHE interventions to communities in and around biodiversity threatened areas in Africa and Asia. The expectation is that USAID/PRH will provide seed funding and technical assistance to these projects in order to leverage USAID Mission funding, through buy-ins, or other donor 26 MID-TERM EVALUATION OF THE BALANCED PROJECT funding to further support these efforts. In addition to this, BALANCED is expected to scale up successful PHE efforts in the Philippines and Madagascar. (All BALANCED activities in Madagascar have been suspended since the 2009 coup). PURPOSE OF THE EVALUATION: The purpose of this activity is twofold: 1. To conduct a mid-term evaluation of the performance of the BALANCED Project to date and provide recommendations to inform the last two years of project implementation. 2. Provide recommendations to USAID to inform the design of a follow-on PHE project which will start after the BALANCED project ends. STATEMENT OF WORK: Mid-term Performance Evaluation of the BALANCED Project (70%): The evaluation should assess whether the BALANCED Project implementers, the Coastal Resources Center, University of Rhode Island, and its partners, Conservation International and PATH Foundation Philippines (PFPI), are successfully achieving the BALANCED Project Objective and Intermediate Results. The evaluation should also identify and describe the key factors leading to the successes and shortcomings of the program. Based on these factors, the evaluation should provide concrete recommendations to the BALANCED and USAID project management team that will inform the last two years of project implementation. Key questions to be answered are: 1. To what extent has the BALANCED project achieved the expected results under each intermediate result: – IR 1: Capacity built for integrated PHE implementation (for expected results, refer to pages 9-12 of the Technical Application (TA), originally labeled IR 2; also see the Performance Management and Reporting Plan (PMP) for descriptions of indicators and targets). – IR 2: PHE knowledge and tools developed, organized, synthesized and shared (refer to pages 6-9 of the TA for expected results, originally labeled IR 1; also see the PMP for descriptions of indicators and targets). – IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity (refer to pages 12-14 of the TA for expected results; also see the PMP for descriptions of indicators and targets). 2. By implementing activities under IR 1, IR 2, and IR 3, to what extent has the BALANCED Project been effective in achieving the Project Objective: ―Advance and support wider use of effective PHE approaches‖? Evidence of achievement of the Project Objective would include, but is not limited to: – Effective PHE approaches are fine-tuned and readily available to donors and implementing agencies. – Increase in the number of organizations adopting and implementing PHE approaches. – A cadre of experienced PHE professionals, including from developing countries, is available to implement PHE programs and provide technical support to new and continuing PHE efforts. – Increased support (financial or otherwise) for PHE approaches beyond USAID. MID-TERM EVALUATION OF THE BALANCED PROJECT 27 3. What are the key factors contributing to the successes and shortcomings of the project? 4. What steps should USAID and BALANCED take to address these factors in the last two years of the BALANCED project? Please identify both immediate and longer term steps. Future Directions (30%): GH/PRH intends to build on the lessons learned during the implementation of the BALANCED Project to design a follow-on award. The findings of the evaluation will inform the design of the follow-on project. We are most interested in learning if the strategy is effective, realistic, and appropriate considering the funding available (assume follow-on will be funded at the same levels as the BALANCED project). Key questions to be answered are: 1. Are there activities under BALANCED that are not contributing as expected to the achievement of the Project Objective? How could these activities be adjusted to have more impact? 2. What are activities, not included in BALANCED, that should be added to the follow-on project in order to achieve the Project Objective? 3. Should the Project Objective be changed for the follow-on? If so, how? METHODOLOGY: Shortly after the consultant is identified, a conference call will be held with USAID/GH/PRH to introduce the main USAID contacts, to address any questions or concerns, and to discuss and clarify the evaluation questions. The consultant should read the RFA, TA and the annual workplans prior to this call. A planning meeting will be held in Washington, DC. This meeting will allow the evaluator to meet with USAID staff to be briefed on the BALANCED Cooperative Agreement, the activities of the BALANCED Project, and future plans. It will also allow USAID to present the evaluator with the purpose, expectations, and timeline of the assignment, and to address any questions. At this meeting, the individuals will: Clarify roles and responsibilities for the evaluator, USAID/GH/PRH, GH Tech, and BALANCED, Review and finalize the evaluation questions, Review proposed data collection methods, interview questions, and the data analysis plan, Review key stakeholders selected for interviews, Review and finalize the evaluation work plan and share with USAID, and Review and clarify any logistical and administrative procedures for the assignment. For this evaluation, the sources of information on the performance of the BALANCED Project include: interviews with key stakeholders, copies of annual work plans and results review reports, semi-annual reports, annual management reviews, and products and documents produced by the BALANCED Project (most documents will be provided electronically). These data sources detail the activities of the project and describe issues complicating implementation and their resolution. Additional information can be acquired by the evaluator through interviews with CRC/BALANCED staff, USAID/Washington staff, USAID/Mission staff, other USAID cooperating agencies, BALANCED partners, other in-country stakeholders and field visit observations. The suggested relevant documents and interviewees are identified below. 28 MID-TERM EVALUATION OF THE BALANCED PROJECT 1. Self-assessment discussion: USAID will request that CRC/BALANCED conduct an informal self-assessment of the BALANCED Project, based largely on the questions in Part 1 of the Statement of Work section above, and be prepared to share their findings during an in person interview with the evaluator. No written self-evaluation is required of CRC/BALANCED, but a Power Point presentation is requested. The results of the self￾assessment and Power Point presentation will be shared with the evaluator during a face to face meeting between the evaluator and CRC staff related to the BALANCED Project. 2. Background Documents/Materials: The following documents will be provided to the evaluator. Other documents may be added or requested as needed. 1) Assessment of USAID’s Population and Environment Projects and Programming Options (2007) 2) Review of Population, Health, Environment Programs supported by the Packard Foundation and USAID (2005) 3) PRH Results Framework 4) PHE Technical Leadership Cooperative Agreement RFA (Appendix A) 5) BALANCED Technical Application in response to the RFA 6) Annual Work plans, Years 1-3 7) Performance and Management Plan (Appendix B) 8) Semi-Annual Reports, #1-5 9) Results Review - Year 2 10) Performance Management Review Reports of the BALANCED project (2009, 2010) 11) List of BALANCED country activities and contact persons 12) List of key stakeholders related to BALANCED activities, including collaborating partners, capacity training participants, users of PHE tools, and seed grant recipients 13) List of all products (training curricula, websites, documents etc.) produced by BALANCED and weblinks to the products. Electronic copies will be provided if weblinks are unavailable. 14) PHE Toolkit (www.k4health.org/phe) 15) Final communications strategy and resource mobilization strategy 16) Advisory Committee notes 17) Cumulative PMP and field indicator summary report for years 1-3 (when completed) 18) List of key informants/interviewees (ranked according to recommended priority) 3. Interviews: The evaluator will interview the USAID BALANCED Management Team (AOTR and TA), other USAID staff, including USAID Mission and Bureau field staff, and BALANCED related staff at CRC, PFPI, and CI headquarter and field levels. The evaluator is also expected to interview other stakeholders related to the BALANCED Project such as cooperating agencies that are working with BALANCED, organizations receiving training or other capacity building from BALANCED, target organizations using the PHE Toolkit, donors, and other parties chosen by the evaluator. In most cases, it is expected that interviews will be conducted in person, by telephone, by Skype, or via videoconference. There will be an opportunity to interview stakeholders in Tanzania, where BALANCED has field activities. A list of suggested interviewees (key informants) at CRC, USAID and other organizations will be provided by CRC and USAID. The highest priority key informants will be identified by USAID and CRC. The evaluator will draft a key informant interview questionnaire and will share with USAID/GH/PRH prior to the planning meeting. The evaluator will be MID-TERM EVALUATION OF THE BALANCED PROJECT 29 provided with contact information, USAID staff will send e-mail of introduction to interviewee (copying evaluator). Scheduling will be done by evaluator. 4. Field Visits: The evaluator will travel to Tanzania to visit ongoing BALANCED activities and interview key informants. The site visit is an opportunity to assess the implementation and results of a PHE field project either implemented by BALANCED or by a seed grant recipient and conduct interviews with key informants in that country to assess BALANCED in-country activities. DELIVERABLES: Before beginning work, the evaluator is strongly encouraged to review the checklist by which the report will be assessed. This is attached as Appendix C, and is also available online at http://www.usaid.gov/policy/evalweb/evaluation_resources.html. 1. Evaluation Work plan, Including Data Collection/Analysis Plan: The evaluator will provide PRH with a work plan for approval by USAID, including a data collection and analysis plan and a key informant interview questionnaire. The general methodology to be used will be reviewed and discussed at the planning meeting. The evaluator will then be responsible for finalizing the overall work plan, developing a schedule for specific activities, and addressing other operational and logistical issues as needed. The data collection plan and questionnaire created will ensure that information collected by the evaluator during all interviews and the country visit is comparable and consistent. These materials should be sent to USAID prior to the planning meeting. 2. Draft and Final Evaluation Report: Outline: The evaluator will provide PRH with a preliminary outline of the report structure before writing begins (electronic version only). Draft report: Analyzing data collected from document review, interviews and field visits, the evaluator will provide PRH with a draft report that includes all the components of the final report (electronic version only). USAID will provide comments on the draft report to the team within three working days. Final Report: The evaluator is required to submit a final internal report (in English) to PRH (two hard copies and an electronic copy) and to submit an electronic copy of the public report to the Development Experience Clearinghouse (http://dec.usaid.gov/index.cfm?p=docsubmit.submitDoc&CFID=892811&CFTOKEN= 45045408). The final report should include, at minimum, the following: executive summary; scope and methodology used; important findings (empirical facts collected by evaluators); conclusions on key assessment questions (evaluators’ interpretations and judgments based on the findings); recommendations for the future of BALANCED (proposed actions for management based on the conclusions); and lessons learned (implications for future designs and for others to incorporate into similar programs). The report should be no longer than 20 pages, excluding annexes. Per USAID Evaluation Policy, this SOW should also be attached as an annex to the final report. As this report is primarily intended for internal USAID use in assessing the performance of the BALANCED Project and defining future program needs, the team will prepare one edited/formatted final report for public release that omits the Future Directions and procurement sensitive information. The Future Directions information will be submitted as a separate well-marked Internal USAID Memo—not using the USAID report format. Once USAID/PRH signs off on the final unedited report for public release, GH Tech will have the document professionally edited and formatted and will provide the final report to 30 MID-TERM EVALUATION OF THE BALANCED PROJECT USAID/PRH for distribution. It will take approximately 30 business days for GH Tech to edit/format and print the final document. The final public report will be submitted to the USAID Clearinghouse and the GH Tech website for posting. 3. Debriefings: The evaluator will provide two debriefings to USAID and CRC in Washington DC. EVALUATOR: One external consultant will be identified and recruited through the GH Tech mechanism to serve as the evaluator. The evaluator should have expertise in the following areas: An advanced degree or commensurate experience in the social sciences, family planning and reproductive health, natural resources management or a related field. Previous experience in program evaluation. Experience in evaluating USAID-funded programs is desirable but not necessary. Familiarity and/or experience in PHE, particularly in sub-Saharan Africa or Asia, is desirable but not required. The evaluator must be able to quickly learn about the PHE community or practice, work closely with USAID staff, evaluate and synthesize information quickly, make clear and well-founded recommendations, and produce clear and useful written reports and verbal debriefings. SCHEDULING AND LOGISTICS: Once the consultant is identified and recruited, the process for document review and interviews with key informants can begin as soon as possible. Ideally, the evaluation will be conducted between May 16 and June 30, 2011. BALANCED project staff will assist the evaluator with in￾country logistics. USAID will not be providing office space or equipment for the evaluator. An illustrative distribution of level of effort (LOE) is seen below. A final time line with distribution of LOE will be developed in consultation with the team at an initial meeting and Team Planning meeting. Contractor Task Total Person-days Document Review 3 Travel to/from DC 2 Meeting with USAID and GH Tech to discuss SOW, clarify roles and responsibilities, identify key stakeholders for interviews, review work plan, and review report outline. 1 Draft and finalize assessment methodology. Develop interview instruments. Finalize work plan. Set up all state-side meetings and plan site visit. 2 Conduct in person, phone, or e-mail interviews with BALANCED staff, partner organizations, USAID, and other stakeholders 6 Travel to/from Rhode Island and meet with BALANCED Project staff 1 Conduct in-depth interviews and site visit in one country 14* Prepare 1st draft of report, review with USAID 9 MID-TERM EVALUATION OF THE BALANCED PROJECT 31 Contractor Task Total Person-days Travel to/from DC 2 Prepare and deliver presentations for USAID and external audiences 1 Revisions and final reports 4 Total 45 * Assumes 2 days for travel to the country, 10 days in country and a final 2 days to travel back to the States. Deliverable Approximate Day Due Draft work plan 3 days prior to the planning meeting Final work plan (only if necessary) 1 day after planning meeting Outline of report structure Before writing begins Draft Report 32 Presentation/debrief TBD (35-40?) Final Report (in two sections) 44 Report submitted to the DEC 44 Days LOE= total The primary point of contact at USAID will be Elizabeth Pleuss; your secondary contact is Heather D’Agnes. CLIENT ROLES AND RESPONSIBILITIES: 1. Provide technical leadership and direction to the evaluator. 2. Provide GH Tech with additional information regarding the project contractors or NGOs evaluated/assessed and information regarding their affiliates to help prevent/identify potential conflicts of interest with proposed consultants. 3. Identify and prioritize background materials for the consultants and provide them to GH Tech as early as possible prior to the assignment. 4. Provide GH Tech with a list of site visit locations early on in the planning process, so we can plan in-country travel and prepare an accurate in country travel line items costs for the budget and key contacts at each site and guidance on the travel calendar (i.e. number of in￾country travel days required to reach each destination, and number of days allocated to interviews at each site). 5. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation) and identify the client points of contact that may assist with logistics (i.e., visa letters of invitation etc.) 6. Provide guidance regarding client staff participation (both mission and USAID/W) in the assignment (i.e., who will participate, how long, source of funding for their participation). 7. Help to arrange appointments with key stakeholders and, if appropriate, accompany the evaluator on these introductory interviews (especially important in high-level meetings). 8. Assist in identifying and helping to set up meetings with local professionals relevant to the assignment, if appropriate. 9. Provide timely review of draft/final reports and approval of the deliverables. 32 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 33 ANNEX B. DOCUMENTS REVIEWED FOR THE EVALUATION The evaluator received a comprehensive set of documents from USAID/Washington on BALANCED. These documents included the BALANCED RFA, BALANCED work plans and management reports, semi-annual reports, and semi-annual reports publications lists. The evaluator requested additional documentation on BALANCED from USAID and also collected during the course of this evaluation a number of supplementary materials, with the most relevant and useful included in the list below. LIST OF BACKGROUND DOCUMENTS FOR BALANCED EVALUATION BALANCED Project Planning and Management Documents PHE Technical Leadership Cooperative Agreement RFA BALANCED Technical Application in response to the RFA Performance and Management and Reporting Plan Communications Strategy Resources Mobilization Strategy Annual Work Plan, Year 1 Semi-Annual Report, #1 Semi-Annual Report, #2 Performance Management Review Report of the BALANCED Project (2009) Annual Work Plan, Year 2 Semi-Annual Report, #3 Annual Work Plan, Year 3 Semi-Annual Report, #4 Advisory Committee notes, 2009 Take Home Messages from Advisory Committee Meeting Results Review—Year 2 Performance Management Review Report of the BALANCED Project (2010) Semi-Annual Report, #5 BALANCED Status Evaluation Chart, Indicator Summary Report BALANCED Tanzania Concept Paper BALANCED Summary of PMP Results—Year 3 BALANCED Year 3 Deliverables Annual Work Plan, Year 4 34 MID-TERM EVALUATION OF THE BALANCED PROJECT BALANCED Project Deliverables and Contacts List of BALANCED key collaborating partners (compiled by BALANCED staff) List of other stakeholders, partners List of BALANCED activities by country List of all products (training curricula, websites, documents etc.) produced by BALANCED and web links to the products. (Link to PHE Toolkit (www.k4health.org/phe). PHE South-to-South Exchange Participants’ Directory USAID Background Information PRH Results Framework PRH Results Framework with indicators Review of Population, Health, Environment Programs supported by the Packard Foundation and USAID (2005) Assessment of USAID’s Population and Environment Projects and Programming Options (2007) PRH PHE Investments Spreadsheet (2011) Additional Documents of Interest Linking Population, Health, and the Environment, in Mount Sinai Journal 78:394-405, 2011 BALANCED Newsletter, June 2011 Healthy People, Healthy Ecosystems: A Manual on Integrating Health and Family Planning into Conservation Projects; Judy Oglethorpe, Cara Honzak, and Cheryl Margoluis; WWF, USAID, Johnson and Johnson 2008 Healthy Families, Healthy Forests, USAID and Conservation International BALANCED Newsletter: PHE Approaches, April 2010 The Importance of Population for Climate Change, PAI Fact Sheet #37 Sharing the Forest: Protecting Gorillas and Helping Families in Uganda; Focus on Population, Environment, and Security; ECSP Wilson Center, Oct. 2008 The Web of Life; Fritjof Capra; A New Scientific Understanding of Living Systems; Fritjof Capra; New York: Anchor Books, 1996 MID-TERM EVALUATION OF THE BALANCED PROJECT 35 ANNEX C. LIST OF ORGANIZATIONS AND INDIVIDUALS INTERVIEWED WASHINGTON, D.C. USAID—Office of Population and Reproductive Health Ellen Starbird, Deputy Director Elizabeth Schoenecker, Chief, PEC Heather d’Agnes, Technical Advisor for PHE and BALANCED Agreement Officer’s Technical Representative Wilson Center Environment Change and Security Program (ECSP) Geoff Debelko, Director, ECSP Meaghan Parker, Writer/Editor, ECSP World Wildlife Fund Cara Honzak, PHE Technical Advisor Population Action International Roger-Mark deSouza, Deputy Director for Research Population Reference Bureau Jason Bremner, Program Director, Population, Health, and Environment (phone interview on June 22nd) Jane Goodall Institute Alice Macharia, Director for East Africa Programs (phone interview on June 22nd) Family Health International Therese Hoke, Scientist, Health Services Research (email response rec’d July 18) BALANCED Project/D.C.: Linda Bruce (CRC), Director Janet Edmond (CI), Deputy Director Team Self Assessment Presentation—Linda Bruce; Janet Edmond, and (by phone) Leslie Squillante BALANCED PROJECT, COASTAL RESOURCE CENTER, UNIVERSITY OF RHODE ISLAND: Brian Crawford, Director International Programs (by phone) Leslie Squillante, Deputy Director Cindy Moreau, Financial Administration Elin Torrel, M&E Don Robadue Bob Bowen 36 MID-TERM EVALUATION OF THE BALANCED PROJECT BALANCED PROJECT, PFPI: Joan Castro, Director Ricky Hernandez, consultant Leona d’Agnes, Board member and consultant USAID/TANZANIA Tim Manchester, Senior Reproductive Family Planning & Health Advisor Gabriel Batulaine, Natural Resource Management TANZANIA COASTAL MANAGEMENT PROJECT, TANZANIA Jeremiah Daffa, Director, TCMP-PWANI Project Juma Dyegula, BALANCED Project Patrick Kajubili, PWANI Project NGO PARTNER ORGANIZATIONS, TANZANIA: Nsaa-Iya Kihunrwa, Program Director, Roots and Shoots, Jane Goodall Institute Steven Kiruswa, Heartland Director, African Wildlife Federation WCS, Iringa Feddy Mwanga, Technical Director and Faith Kasulwa, Program Officer, ACQUIRE, EngenderHealth Giulia Basana, Monitoring and Evaluation, Pathfinder Grace Lusiola, AIDSTAR, JSI Joyce Dakaru (Acting Director); Edward Kpande (Finance Officer); Gabriel Sundakare; and Peter: LOOCIP/Longido District, Arusha Partner Organizations and Government Officials—Bagamoyo, Pangani, and Tanga: Hon. Zipporah Pangani, Pangani District Commissioner Rashid Neneka, Pangani District Executive Director Dr. Ole Sepere, Reproductive and Child Health, Pangani District Hospital Dr. Dunstan Pondah, Pangani District HIV/AIDS Control Coordinator Joseph Makombe, Pangani District Health Officer Dr. Ally Uredi, Regional Medical Officer, Tanga Region Doroth Lema, Tanga Regional Reproductive and Child Health Coordinator George Magige, Regional Representative, PSI-Tanga Dr. Vera Pieroth, Executive Director and Abdullah Mfuruki, Communication Manager, UZIKWASA (Uzima Kwa Sanaa (LIVE FOR ARTS)) BALANCED Trainees and Participants from the Wards and Villages of Bagamoyo District Mkange Ward in Bagamoyo district– discussion with 29 members of Uwamke (Wake-Up) SACCO (registered micro-credit association) Saadani Village in Bagamoyo—discussion with 9 SACCO members and bread makers (bakery) MID-TERM EVALUATION OF THE BALANCED PROJECT 37 BALANCED Trainees and Participants from the Wards and Villages of Pangani District Buyuni Village in Pangani district—CBD Mwanaisha Mkalamo Ward in Pangani—Interviews with PE Rukia Sefu and CBD Mkejina Issa (saw fuel cooking efficient stove); observed Theater for Development troupe PHE skits; group discussion with CBDs and PEs; met with Anna-Mkalamo dispensary nurse Mkwaja village in Pangani district—discussion with CBDs Mikocheni Village in Pangani—meeting with local government executive officer and saw shop of PHE provider Abdallah Masingano Sakura Village in Pangani—group discussion with GIZ CBDs-trained by BALANCED on PHE approach, BALANCED-trained PEs, and SACCO members, ADDO (medical store) OTHER EXPERTS, PARTNERS, AND DONORS: USAID/Philippines Rebecca Guieb, OEE/Manila (email response received 7/21) Maria Teresa Carpio, OPHN/Manila (email response received 7/28) Pathfinder International: Sono Aibe, Senior West Coast Representative for Strategic Initiatives Packard Foundation: Sahlu Haile, Senior Representative for sub-Saharan Africa Yemeserach Belaynay, Foundation Representative for Ethiopia Bernd Cordes, Foundation Program Officer, Conservation Program Environmental Leadership Program, University of California Berkeley Robin Marsh, Co-Director Evaluation and Research Technologies for Health (EARTH) Inc. Lynn Gaffikin, President (and Advisory Group for BALANCED) John Snow, Inc Nancy Harris, Vice President (and Advisory Group for BALANCED) Johns Hopkins/CCP Guy Chalk, IT Manager, Knowledge for Health Project (by email) OTHER PARTNERS AND EXPERTS: Ethiopia PHE Consortium Negash Teklu, Executive Director CRC Ghana, Integrated Coastal and Fisheries Governance Initiative, Western Region, Ghana Mark Fenn, Program Director 38 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 39 ANNEX D. INTERVIEWS AND FIELD VISITS BALANCED EVALUATION (TANZANIA) June 25th-July 10th June 25th- 28th—Dar es Salaam Orientation and Initial Interviews Orientation and Scheduling Meetings with BALANCED (Juma Dyegula and Ricky Hernandez) Informal meetings with USAID Interviews with Pathfinder and Jane Goodall Institute staff who had participated in BALANCED training programs Initial interviews with TCMP/BALANCED staff (Juma Dyegula) and PFPI consultant for BALANCED (Dr. Ricky Hernandez) June 29th—BALANCED sites in Bagamoyo and Pagani Mkange Ward: Discussion group with 30 members of Uwamke SACCO Saadani Village: Small discussion group on lending and FP with 8 SACCO members Buyuni Village: Interview with CBD Mwanaisha June 30th—BALANCED sites in Pagani Mkalamo Ward: Interviews with PE Rukia and CBD Mkejina; performance by Theater for Development, local theater troupe; discussion with SACCO members; observe fuel-efficient stoves and bread baking stoves; discussion with Anna Meshack, nurse in charge of government dispensary; visit of weekly market Mkwaja Ward—discussion with CBDs Mikocheni Village—meet village executive officer and visit shop of Peer Educator Abdulah Sakura Village—group discussion meeting with PEs and CBDs; visit local ADDO owner Asha Salim July 1st—Pagani Meetings with District Commissioner, Zipora Pagani and District Executive Director Rashid Neneka Interview at Pagani District Hospital with Dr Ole Sepere (Reproductive and Child Health), Dr. Dunstan Pondah (District HIV/AIDS Control Coordinator), and Joseph Makombe (Health Officer) Visit and discussion with UZIKWASA director Dr. Vera and communications officer Adulah July 1st—Tanga Meeting with Regional Medical Officer Dr. Ally Uredei and Regional Reproductive and Child Health Coordinator Doroth Lema Meeting with PSI Regional Director GIZ, MoH and PSI and then travel to Arusha by road from Tanga (TCMP driver, Ricky and Don) July 2nd—Travel to Arusha 40 MID-TERM EVALUATION OF THE BALANCED PROJECT July 4th—Meeting with African Wildlife Foundation and interviews with AWF staff trained and/or in contact with BALANCED July 5—Field visit to Longido District; meeting with local community-based organization LOOCIP and interviews of LOOCIP staff trained who have participated in BALANCED training programs—Peer Education TOT in Dar and PHE Design Workshop in Arusha July 6—Air Travel from Arusha to Dar es Salaam July 7—Interviews with USAID/Dar officials (Gabriel Batulaine, NRM, and Tim Manchester, PRH); interview with Grace Lusiola, AIDStar Project, John Snow, Inc. July 8—Interview with TCMP director Jeremiah Daffa and EngenderHealth ACQUIRE staff, Feddy Mwanga, Sarah Kinyanga, and Faith Kasulwa July 9—Interview with Joan Castro, PFPI Executive Director and Technical Lead for BALANCED (by Skype); debrief meeting with Ricky Hernandez, PFPI consultant July 10-11—Return flight to U.S. MID-TERM EVALUATION OF THE BALANCED PROJECT 41 ANNEX E. KEY INFORMANT QUESTIONNAIRE KEY INFORMANT INTERVIEW QUESTIONS: BALANCED EVALUATION Legend: Sow Questions Italicized BACKGROUND QUESTIONS: B1 How did you come to this point in your career, significant responsibilities over/within … (e.g., a project that seeks to advance PHE)? BALANCED GENERAL QUESTIONS: BG1 Please describe a typical work day for you. a. What kinds of things do you do on a fairly regular basis, including those apart from BALANCED? b. What kinds of things do you do on a fairly regular basis in relation to BALANCED, and specifically IR 1, IR 2, and/or IR 3? BG-2 What in your previous professional experiences most prepared you for implementing this project? BG-3 What advantages and disadvantages have there been as a result of the main implementing partner for a USAID cooperative agreement being a university? BG-4 How has the project dealt with the complexity and built-in challenges of key staff being both part time and based in different locations and time zones? BALANCED INTERMEDIATE RESULTS QUESTIONS: 1. To what extent has the BALANCED project achieved the expected results under each intermediate result: – IR 1: Capacity built for integrated PHE implementation (for expected results, refer to pages 9-12 of the Technical Application (TA), originally labeled IR 2; also see the Performance Management and Reporting Plan (PMP) for descriptions of indicators and targets). 42 MID-TERM EVALUATION OF THE BALANCED PROJECT INTERMEDIATE RESULT 1: BUILD CAPACITY 1:BIR1-1 BALANCED seeks to strengthen PHE capacities … a. How does it do this? b. What has its impact been to date? c. What challenges has it faced? 1:BIR1-2 Describe and weight the mix through which BALANCED seeks to strengthen and institutionalize local capacities for implementing PHE field projects. a. What has its impact been to date? b. What challenges has it faced? 1:BIR3-3 Have there been changes in PHE capacity within countries, such as Ethiopia, Tanzania, and the Philippines, since October, 2008 when BALANCED got under way? a. To what extent can any change be directly attributed to BALANCED and how did this come about? b. What was its contribution? 1:BIR1-4 Part of BALANCED places emphasis on developing and supporting champions for PHE. a. To what extent can the emergence of PHE champions be directly attributed to BALANCED and how did this come about? b. What were BALANCED’s specific contributions? c. Should/could BALANCED have done more to create and support PHE champions? 1:BIR1-5 During the last twelve months how often and for what purposes have you traveled from the office to provide technical assistance or support of some kind? a. Focusing on what you would consider the most successful occasion when you worked in the field, please provide details. b. Concentrating on an assignment to the field that was not optimally successful, please analyze what in retrospect could have been done differently. INTERMEDIATE RESULT 2: KNOWLEDGE MANAGEMENT IR 2: PHE knowledge and tools developed, organized, synthesized and shared (refer to pages 6-9 of the TA for expected results, originally labeled IR 1; also see the PMP for descriptions of indicators and targets). MID-TERM EVALUATION OF THE BALANCED PROJECT 43 1:BIR2-1 What readings or scholars are you familiar with who have best informed and stimulated your thinking about how population, health, and environment may be interlinked or interactive? Should economic enterprise or livelihoods not also be a part of the interactions that occur and are interlinked at the community level? 1:BIR2-2 What do you expect BALANCED to provide in terms of global leadership, knowledge building, and collaboration? a. What are the global leadership strengths to date? b. What are the global leadership weaknesses or shortcomings and how may these be improved? 1:BIR2-2 At the global and/or regional level, to what extent and how has BALANCED helped advance PHE, supported advocacy efforts, and developed other kinds of collaborations with partners? a. With which organizations has collaboration been most effective and/or groundbreaking? b. Have there been missed opportunities for collaboration or areas where more needs to be done? c. In what ways has BALANCED been pro-active in helping global partners strengthen their own PHE programs? 1:BIR2-3 Cite specific examples of best practices, useful approaches, and/or tools drawn from previous experiences within PHE that have been adapted by BALANCED. a. Which do you identify as the most useful and likely to be most widely used? Where have these been described? b. Have these actually been applied within field activities directly supported by BALANCED? c. What other best practices is BALANCED learning about that it intends to apply? 1:BIR2-5 Do you know of any tools or toolkits that you or others have found helpful for understanding how to develop and implement integrated PHE field projects? If so, who developed these, why are they useful, and where can they be found? BIR-3 How often and for what purposes do you yourself use the BALANCED website? a. In addition to the use-data about the website, what other information (anecdotal or otherwise) do you have that the BALANCED website is used and useful for practitioners of PHE? b. What more is planned to increase relevance and usage of this website or other means to reach a wide technical audience? 44 MID-TERM EVALUATION OF THE BALANCED PROJECT INTERMEDITATE RESULT 3: FIELD PROJECTS IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity (refer to pages 12-14 of the TA for expected results; also see the PMP for descriptions of indicators and targets). 1:BIR3-1 Another of the BALANCED Project components focuses on implementing PHE projects on the ground. In addition to providing ongoing support to IPOPCORM in the Philippines, the project also supports coastal and uplands projects in Tanzania and Ethiopia. What do you know about BALANCED successes or shortcomings at this level? 1:BIR3-2 What do you expect BALANCED to provide in terms of country-level field projects? a. What have been some outstanding successes to date? b. Where have there been significant shortcomings or failures? c. What lessons have been learned? 1:BIR3-3 Within countries where BALANCED operates, has the terrain been mapped for ongoing and/or potential PHE activities? 1:BIR3-4 With which countries has BALANCED enjoyed the most success during the first three years? a. Which in-country partners have achieved the greatest successes or have the most potential? b. Have some anticipated or unanticipated country-level partners experienced an expanded or diminished role during the course of BALANCED? c. Where and with which organizations do you wish that BALANCED had been able to do more? 2. By implementing activities under IR 1, IR 2, and IR 3, to what extent has the BALANCED Project been effective in achieving the Project Objective: “Advance and support wider use of effective PHE approaches”? Evidence of achievement of the Project Objective would include, but is not limited to: – Effective PHE approaches are fine-tuned and readily available to donors and implementing agencies. – Increase in the number of organizations adopting and implementing PHE approaches. – A cadre of experienced PHE professionals, including from developing countries, is available to implement PHE programs and provide technical support to new and continuing PHE efforts. – Increased support (financial or otherwise) for PHE approaches beyond USAID. MID-TERM EVALUATION OF THE BALANCED PROJECT 45 2POIR-1 What specific tools, activities, and/or approaches has BALANCED used with regularity and success to advance PHE? 2POIR-2 Are there other activities and approaches that BALANCED could have usefully invested in to support champions and leaders for PHE? 2POIR-3 What have been the successes and shortcomings around leveraging funds or in-kind contributions for PHE from USAID/Missions, other USAID partners, other NGOs, and other funders? What could have been done differently to improve things? 2POIR-4 Would BALANCED or PHE have advanced further if the project had done more earlier with field implementation rather than managing knowledge? 2POIR-5 Overall, has BALANCED been responsive to the field’s technical needs? a. What is the skills mix within BALANCED staff and is it sufficient to respond to field needs for technical and managerial support? b. In what areas can BALANCED improve its response to field needs? c. Does BALANCED have more requests for TA than it can fulfill? In what areas? 2POIR-6 What types of technical inputs does BALANCED provide to other USAID projects and organizations with which it collaborates? a. Please provide examples of specific inputs provided by BALANCED and assess their relevance and technical soundness. b. Are there also some examples where technical assistance has fallen short of increasing the capacity of an organization? c. Are there examples of collaborations that did not develop or continue because what BALANCED could provide did not or was not perceived to match what was needed? Please elaborate. 2POIR-7 What opportunities are there to advocate for and leverage funding from different sources and at different levels? a. Give examples of successes increasing in-country support to advance PHE. b. Give examples and opportunities for increasing support for PHE. 46 MID-TERM EVALUATION OF THE BALANCED PROJECT 2POIR-8 Please describe your role and/or your understanding of how the PPM indicators were developed. a. How would you describe the indicators that are now tracked: input, output, outcome, or other? b. What other indicators could one envision as important to track over the long term to measure achievements of a project like BALANCED? 3. What are the key factors contributing to the successes and shortcomings of the project? 3:POIR-1 What from your point of view has been the most effective part of BALANCED? What is the least effective? 3:POIR-2 How does BALANCED currently build capacity within organizations—through subcontracts, technical assistance, training, staff in residence, other approaches? Please provide examples where this has proceeded well and not so well. 3:POIR-3 Are there examples where knowledge management and field implementation of PHE has not yielded results hoped for? 3:POIR-4 What are BALANCED’s unique contributions or special capabilities for advancing PHE? 3:POIR-5 Are there near-term future opportunities for leveraging funds and/or advancing PHE? 3:POIR-6 What contributions/expectations does your part of the management structure or project have to make to supporting BALANCED and how do you go about doing this? 3:POIR-7 What are the high and low points of working with or within BALANCED since this project started in late 2008? Please provide concrete examples of each? Were lessons learned from these experiences, and (if so) how were they shared with others? 3:POIR-8 What during the last year or over the course of BALANCED are you personally most proud of having done with or within BALANCED? What do you have regrets about having done or not done that would have contributed to BALANCED? MID-TERM EVALUATION OF THE BALANCED PROJECT 47 3:POIR-9 What technical and managerial support does BALANCED receive on a regular basis from USAID? Does BALANCED need more or less of this support? a. Please give specific examples of where support provided by USAID has been most helpful to achieving the BALANCED project objective and IRs. b. Where has USAID’s assistance and support fallen short or worked against BALANCED being effective? 4. What steps should USAID and BALANCED take to address these factors (shortcomings) in the last two years of the BALANCED project? Please identify both immediate and longer term steps. 4:BF-1 In the remaining two years, what would you like to change either about the project or about your own work? 4:BF-2 At this midpoint in implementing BALANCED, are there any significant adjustments that you would like to consider? Are there adjustments within the current management structure or BALANCED approaches that would be helpful? 4:BF-3 What are the most useful and fulfilling parts of your work? What makes you frustrated or unsatisfied? Do you have suggestions for improving the project? What are these? 4:BF-4 In your view, is the BALANCED project objective the right one and are the IRs rightly ordered and weighted (funded)? Are the skills among staff the right ones to achieve these? BALANCED FUTURE QUESTIONS: 1. Are there activities under BALANCED that are not contributing as expected to the achievement of the Project Objective? How could these activities be adjusted to have more impact? BF1-1 What challenges and contradictions are there in fielding an approach that embraces ―advance and support wider use of effective PHE approaches‖? How does BALANCED balance such long￾term ambitions with the relatively small amount of resources in this project? 2. What are activities, not included in BALANCED, that should be added to the follow￾on project in order to achieve the Project Objective? BF2-1 What activities and/or approaches, not included in BALANCED, should be added to the follow￾on project in order to achieve the Project Objective, ―Advance and support wider use of effective PHE approaches‖? 48 MID-TERM EVALUATION OF THE BALANCED PROJECT BF2-2 In your experience, what are the best PHE field projects that you have seen or have supported? What elements within these projects or what aspects of their implementation account for their high standard of excellence? In other words, what are some of the secret ingredients to success? 3. Should the Project Objective be changed for the follow-on? If so, how? BF3-1 If the project were starting anew, what would you change, structurally as well as substantively? BF3-2 If you were the director of BALANCED, or of USAID, what would you change about this project, USAID’s approach to PHE, or USAID itself? BF3-3 Much of the knowledge management seems oriented towards accumulating and disseminating practical how-to knowledge about implementing PHE projects. Is or should there also be effort to build knowledge by investing in theory or understanding about how population, environment, and health are linked? BF3-4 How do you envision PHE 10 years from now in the level and type of countries where BALANCED now operates? BF3-5 What question that I haven’t asked of you, would you wish that I had asked? And what is your response or responses to that/those question(s)? TANZANIA COUNTRY-LEVEL QUESTIONS: TAN-1 Please provide a brief narrative history of conservation efforts, health programs, as well family planning and contraceptive use in Tanzania. a. What have been the ups and downs, successes, and failures in advancing conservation, health, and family planning? b. What is the current status of PHE or integrated approaches at the community level in Tanzania? TAN-2 Would you provide examples of how often and for what purposes you have been in contact with the BALANCED staff? a. Please describe the situation that occurred when contact was made with either BALANCED and/or the USAID mission regarding PHE. b. Was any further assistance from BALANCED provided and what have been the results of that? TAN-3 How and with what effectiveness has the BALANCED Project worked in Tanzania over the last three years? MID-TERM EVALUATION OF THE BALANCED PROJECT 49 a. What has the USAID BALANCED Project provided to directly advance PHE programs in Tanzania? b. Outside of direct financial or technical support to Pangani, what other support has BALANCED provided during the last three years? c. Has BALANCED advanced, hindered, or been neutral with regard to family planning progress in the country? TAN-4 Have you had technical assistance from BALANCED? How often and to do what? What, if any, lasting impact has this assistance had on your organization and its ability to implement PHE? Give specific examples of when this has worked well and when this hasn’t. FIELD SITE QUESTIONS: B3 What is the name of this community, what assistance and funding did it have from TMPC, and what if anything was added by BALANCED? 1:BIR1-5a During the last twelve months, how often and for what purposes has this community received some training or other assistance from BALANCED to advance PHE? 2POIR-2 Has BALANCED identified and supported champions and leaders for PHE? 2POIR-5 Overall, has BALANCED been responsive to the community’s needs? a. What is the skills mix within BALANCED staff and is it sufficient to respond to field needs for technical and managerial support? b. In what areas can BALANCED improve its response to field needs? c. Does BALANCED have more requests for TA than it can fulfill? In what areas? 2POIR-7 What opportunities are there to continue with these activities after BALANCED is no longer here? 3:POIR-1 What from your point of view has been the most effective part of support for this community? What is the least effective? 3:POIR-8 What during the last year or over the course of BALANCED are you personally most proud of having done within this community? What do you have regrets about having done or not done that would have contributed to BALANCED? 50 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 51 ANNEX F. SYNTHESIS OF ANSWERS TO FOUR SOW QUESTIONS 1. TO WHAT EXTENT HAS THE BALANCED PROJECT ACHIEVED THE EXPECTED RESULTS UNDER EACH INTERMEDIATE RESULT: – IR 1: Capacity built for integrated PHE implementation – IR 2: PHE knowledge and tools developed, organized, synthesized, and shared – IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity The indicators tracked on the PMP provide the latest update on progress made within each intermediate result and two strategic objectives. The latter, stipulated by BALANCED within the PMP, are: SO 1—Number of target organizations incorporating PHE tools, protocols, etc., into their work, and SO 2—Dollar value of funds leveraged from USAID Missions and non-U.S. Government sources to support PHE implementation and scale ups. Comparing expected targets and actual achievements, BALANCED is ahead on some indicators, behind on a number, and just about up to date on others. Some of the indicators on which BALANCED lags behind are significant ones, such as amount of funds leveraged or numbers of field implementation sites contributing to knowledge building or success stories. The synthesized response to this SOW question will neither repeat nor refute what is contained in the PMP report. (The latest BALANCED tracking sheet on progress toward indicators may be found in Annex H.) Instead, the response to this question will concentrate on highlights and insights in relation to each IR. IR 1: Capacity Building Training has been the most widely used approach within BALANCED to develop capacity for PHE. Most innovative has been the project’s consistent effort to include a range of individuals and organizations within training programs that also serve the purpose of providing needed skills for those working within seed grants and field projects. These trainings are intentionally set up to serve a ―dual purpose‖: providing understanding of some aspect of PHE as well as specific skills for implementation. This has often been accomplished by simultaneously conducting content training in conjunction with training of trainers. Perhaps another aspect of such ―dual purpose‖ training is that the courses intentionally meet immediate capacity-building needs among key people within field projects as well as develop such capacities among others for near￾term application within ongoing or subsequent work. BALANCED purposefully sought to include in trainings staff from family planning and environmental organizations from within the country where the training was taking place as well as staff from other countries. In addition, BALANCED places importance on providing follow up, such as mentoring or technical assistance in the post-training period. However, though some follow up is provided, insufficient staff or consultant time is available for this to make a meaningful difference. According to a number of key informants who had participated in one of several training opportunities held in Tanzania, the best outcome from including individuals from other countries or organizations in trainings is that they receive a solid orientation to PHE, rather than specific skills to implement this approach. Pre-training, which BALANCED also features as part of its comprehensive approach to building capacity, is also limited, largely due to the same constraint, insufficient staff time. As noted by a key informant who has facilitated some of these trainings, sometimes the right people do not attend. Given the limitations of staff time, with what effectiveness were these dual purposes actually fulfilled? Among those with near-term opportunities to apply the learned skills to implementation, this approach to training seems to have worked quite well. While the evaluator 52 MID-TERM EVALUATION OF THE BALANCED PROJECT was only able to look in any depth at the specific case of Tanzania and the Tanzania Coastal Management Program, those trained were subsequently able to conduct community-level trainings on their own toward implementation of the PHE project. Furthermore, in group discussions with the evaluator, the beneficiaries of such subsequent trainings were able to accurately articulate aspects of the PHE approach as well as describe their own particular roles within its field implementation. These secondary trainings often occurred without direct oversight or assistance from the BALANCED staff member who had provided the initial training. For the other category of those trained, those more distant from field implementation, BALANCED has imparted a degree of interest or basic understanding of PHE more than it has developed capacity for its implementation. Those interviewed in Tanzania who had previously participated in a BALANCED training generally recalled this as a positive learning experience. They were particularly complementary about the trainers and the participatory approach to training that they used. Most trainings also included opportunities to practice skills that had been taught. However, none of those interviewed reported that participating in the particular training had a direct impact on how they then did their work. While they appreciated exposure to new specific skills and understanding, without a near-term opportunity in their ongoing work to use what they had learned, they could not report that the training resulted in capacity building. However, training and trainees did provide BALANCED with new connections to organizations with which they could potentially partner for PHE. Almost all subsequent efforts to extend PHE efforts in Tanzania, such as with AWF and WCS, came about because of initial contact made with individuals from those organizations within BALANCED training programs. Beyond training, BALANCED also provides technical assistance to build capacity for PHE. This has occurred in a number of countries, including Ethiopia, Ghana, Kenya, Tanzania, and Zambia. At times, BALANCED is pro-active in providing such assistance, such as a workshop it conducted on PHE design in Arusha for AWF and some of AWF’s partner implementing organizations. On other occasions, BALANCED has responded to specific invitations for technical assistance, such as occurred when the BALANCED/PFPI East Africa consultant participated in the planning meeting organized by Pathfinder and The Nature Conservancy for PHE in Mahale, Tanzania. In most instances that have occurred over the last two years, the East Africa consultant has been the primary BALANCED presence, serving as a consultant to PFPI for both training and technical assistance. For other countries and for East Africa earlier in the project, others from PFPI have provided the training and technical assistance. Some exceptions are when the project director facilitated with the PFPI consultant an information, education, and communication TOT held in Dar es Salaam. A CRC consultant also held a designing for behavior change training in the TCMP project area. While the former received positive recall responses from those interviewed, according to several reports the latter did not serve on-the-ground needs for PHE in the TCMP project area. Another avenue for developing capacity in PHE has been the support that BALANCED has provided to leaders and champions for PHE. The principal BALANCED activity in this regard has been to develop and disseminate stories about particular individuals who are playing unique, interesting roles in PHE. These stories are developed by BALANCED staff and disseminated from the PHE website (see http://www.k4health.org/toolkits/phe/nurturing-phe-champions). These stories are also co-listed on the Wilson Center website. Some stories are also featured in the BALANCED newsletter or disseminated through other distribution channels, including once within the USAID publication Frontlines. On occasion, notably at the Uganda Family Planning Best Practices Conference in October 2009 and the similar Senegal Conference in November 2011, BALANCED has been instrumental, in collaboration with PRB, in convening speakers and sessions on PHE. Presenting at such MID-TERM EVALUATION OF THE BALANCED PROJECT 53 international venues is useful for PHE leaders, such as Negash Teklu, the executive director of the Ethiopia PHE Consortium, in reaching a wider audience on the PHE work they actively support. These opportunities also serve to help these leaders develop their public presentation skills. One wonders, however, if PHE would not have been better served by having such leaders present at international conservation rather than at family planning conferences. Co-sponsoring Gladys Kalema of Conservation through Public Health to attend a Millennium Development Goals meeting in New York, where she talked about PHE in Uganda, was one such effort on the part of BALANCED to reach a wider audience. Emerging leaders for PHE have in addition sometimes received other support from BALANCED. The most recent example of this is partial support provided for a TCMP staff member involved in livelihood development in Pangani and Bagamoyo to attend the Environmental Leadership Program at the University of California, Berkeley. On his return to Tanzania, he stopped off in Washington, D.C., to give a presentation on his work at the Wilson Center. One further BALANCED-supported activity is also noteworthy in the context of capacity building: the study tour to the Philippines to learn about PHE. Participants from several African countries joined for this week-long event, with a number reporting it as a high point for gaining new understanding about PHE. This event, which occurred fairly early in the project, provided a solid basis for expanding PHE interest and activities in East Africa. Overall, BALANCED has effectively used training to build capacities for PHE within field implementation sites. Given the small number of field projects actually being implemented, capacity development has been correspondingly limited. In these instances, those trained were actually able to apply what they had learned, thereby solidifying newly acquired skills. Technical assistance has been a useful adjunct for developing capacities. However, efforts to do so have been limited due to funding. Investments to support leaders and champions have not been nearly as extensive as those for training. However, this has been useful for those who received direct attention in the form of featured stories or support for participating in conferences. IR 2: Knowledge Management The most widely recognized success of BALANCED, particularly among those interviewed in the U.S. and those with regular Internet access, is the compilation of PHE materials on the K4Health website (see http://www.k4health.org/toolkits/phe). This was accomplished in close collaboration with the Johns Hopkins Center for Communication Program, which reported great appreciation for the level of expertise that CRC brought to this effort. Much of what is on this website is material migrated, after considerable effort on the part of BALANCED, to identify those most relevant and useful, from the less rigorously assembled website of a previous USAID project, the Environmental Health Project. Furthermore, most of the more up-to-date materials populating this new website come from previous programs and projects in PHE. With the exception of stories on PHE leaders and BALANCED newsletters, the website contains previous rather than recently developed reference materials for PHE. Tools or guidelines emanating from BALANCED itself have been long delayed in appearing on this website; the four tools and guidelines in development have undergone extensive revision and editing. They are scheduled to be finalized, however, under the Year 4 Work Plan. Thus, this IR has been aptly named ―knowledge management‖ rather than ―knowledge building.‖ In response to direct questions about the need for the latter, most knowledgeable key informants did not consider extensive work for deeper-level, even theoretical, understanding of PHE as within the purview of BALANCED. Despite its university base, CRC and BALANCED are viewed as engaged in developing practical approaches for implementing PHE rather than delving into theoretical explanations or expositions about its underpinnings. Clearly, some BALANCED staff members have capabilities along these lines, as evinced by some of the 54 MID-TERM EVALUATION OF THE BALANCED PROJECT conceptual frameworks they have worked on (e.g., a diagram in the most recent newsletter depicting livelihoods interrelations with PHE). As some key informants explained, linkages to academics developing more theoretical perspectives on PHE is more the province of the Wilson Center than a project like BALANCED. Thus the website and its development has aimed for and has to some extent achieved a one-stop website containing practical guides and reports on how PHE may be done. One way for the website to achieve a higher level of success would be to build it out as a virtual community of practice around PHE. The website does not at this point have features, such as up-to-date announcements of relevance to PHE or timely webcasts, to bring users back on a regular basis. In addition, it is not clear that the site is used regularly and extensively even by those most involved in PHE. Many key informants in the U.S., and even a number within BALANCED itself, reported that they do not often go or have never been to this site. They are aware that it is there, but often have a level of expertise about PHE to which the website contributes little in addition. Listserv emails when something is available were appreciated by those who received them. However, the website has not yet been sufficiently embraced by the PHE community as a place to post materials to share with other practitioners, actual as well as potential. For those interviewed in countries where BALANCED works, reported regular usage was even lower. Connectivity and electricity shortages were mentioned as factors contributing to this non-use. The recent connection made between the K4Health PHE site and the Ethiopia Consortium’s website was cited by those knowledgeable of both as a positive development. CRC technical assistance was essential to making this connection. IR 3: Field Implementation of PHE Among many other indicators, the PMP counts and tracks the ―number of BALANCED field site activities that produce results that can feed into KM activities (development of tools and success stories).‖ To date, field implementation itself remains a small number, an under-achievement. Clearly, the project did not give as much priority to this early on as it did to activities under the other two IRs. The logic in this escapes the evaluator. USAID/PRH’s dual emphasis that seed grants were for testing/piloting family planning and needed to be supplemented by other funding sources may have led BALANCED to allocate significantly more funding for IR 2 than IR 3. However, given USAID/PRH restrictions, BALANCED could have opted to front-load spending under IR 3 and/or done more under IR 1 capacity building to directly support field implementation. While early efforts to develop and populate the website could clearly have occurred in parallel with greater effort to support field-based activity, the project’s success was ultimately and largely to be vested with new field applications of PHE and scaling these up through leveraged funds. At this point in the project, there seems little opportunity for achieving such broad-level success as a result of on-the-ground field implementation. The Philippines, with one significant buy-in from the Mission already under way and another in the works, will stand as the exception—scale up into several new regions based on a solid earlier PHE experience. PFPI deserves much credit for this. If the second buy-in comes through in a timely fashion, all three participating partner organizations will have the opportunity to work on field-level implementation within this exciting and interesting PHE scale-up context. The Philippines will be a defining moment for BALANCED: implement a comprehensive PHE model by working and learning together, or work separately on PHE approaches unique to each of the three organizations. Does BALANCED over these last two years become a project within which three organizations complement each other—or do the partners remain competitors? MID-TERM EVALUATION OF THE BALANCED PROJECT 55 Field-level implementation has progressed slowly in the Coral Triangle and Africa regions. As reported by a knowledgeable key informant, the Coral Triangle Initiative itself has been slow and cumbersome about moving toward field implementation. Thus, what could have been a useful mechanism within which BALANCED could have worked has not yet panned out. Efforts to identify potential field implementation sites, particularly in Papua New Guinea, still hold some promise. However, with limited allocation for seed grant funding, some difficult decisions about which will be funded are likely. Both CRC and CI brought to BALANCED years of experience working in Africa. Unfortunately, Madagascar was derailed early on by political events. Also, changes in USAID/Nepal funding priorities resulted in ending PHE efforts that had started in that country. Nevertheless, BALANCED does support some interesting experimental pilots in other countries. In Africa, CRC’s 15 years of involvement with TCMP served to jumpstart PHE within a coastal area of Tanzania. The Jane Goodall Institute’s (JGI) previous work with integrated approaches, including family planning, in Gombe could also have been something to build upon. However, for reasons not clear to the evaluator, such a relationship did not develop beyond JGI staff participation at three training programs. Ethiopia also had a well established and even more extensive history with PHE as a result of Packard Foundation investments. BALANCED was able to build on this, as well as on its work with the Ethiopia Consortium for PHE, to fund an interesting uplands project with Ethio Wetlands and Natural Resources. Other field implementation projects subsequently supported have similarly come about opportunistically. CRC’s work with fisheries in coastal Ghana opened the way for promising assistance from PFPI for PHE design. Connections made with WCS provided an avenue for inserting family planning into the Community Markets for Conservation (COMACO) project in Zambia. Recently a call for PHE proposals yielded a large number of responses. One wonders why such a call was not issued far earlier in the project. 2. BY IMPLEMENTING ACTIVITIES UNDER IR 1, IR 2, AND IR 3, TO WHAT EXTENT HAS THE BALANCED PROJECT BEEN EFFECTIVE IN ACHIEVING THE PROJECT OBJECTIVE: “ADVANCE AND SUPPORT WIDER USE OF EFFECTIVE PHE APPROACHES”? The project was designed by USAID/PRH with three IRs, each to contribute to the overall objective. Achievements by BALANCED within each have lagged behind what they could have been. In addition, it is not clear that the emphasis given within each IR best served the purpose headlined in the project objective. As many key informants within as well as outside of BALANCED have reported, the project lacks a unifying vision and strategic plan against which to prioritize and map activities under the IRs. This omission has been significant. Demonstration that BALANCED fully understands the linkages, potential and actual, among the three IRs would have gone a long way toward positively answering this question. Clearly, there is a basic understanding that there are linkages among the three IRs to build on. However, in neither the RFA nor the proposal was extensive exposition provided on such interconnections among IRs and their interrelations with the project objective. To trace one example, as stated in the RFA, KM can build upon what is used and learned during implementation of field projects. However, KM (IR 2) received higher priority and greater effort early on than did field implementation (IR 3). USAID/PRH takes some responsibility for this allocation. The combination of restrictions on use of central funds for field implementation and encouragement to leverage supplemental funding for field implementation underlies guidance provided by USAID/PRH. The Year 4 budget shows this to still be the case. In reality, USAID/PRH restrictions on IR 3 funding affected allocations among IRs. On the one hand, if 56 MID-TERM EVALUATION OF THE BALANCED PROJECT seed grants were to be used primarily to promote and provide family planning, BALANCED needed to counterbalance that by seeking complementary funding early in the project. Identifying Missions where supplementary support would be available would have been a useful approach. On the other hand, a broader interpretation of how central funds may support field implementation could have been helpful. That seed grants could also be used to test or pilot approaches involving family planning could have been interpreted to mean implementing variants of PHE that are more than FP-led or layered. Because BALANCED has not articulated what its strategy is for guiding implementation and setting priorities among the many activities it could implement, it is not possible in this instance to determine the logic behind giving priority to KM over field implementation. This makes it appear that BALANCED leadership considers a website collection of materials on PHE as the most direct pathway to achieving the project objective. A priori, the evaluator does not think that the case. Similarly, many of BALANCED’s efforts to build capacity for PHE (IR1) have not been sufficiently extensive or rigorous so as to meaningfully contribute to advancing PHE. One-off training exposures can at best orient participants to PHE, not enable them to practice it. Though PHE website awareness is promoted within training programs, almost no trainees reported subsequently accessing the website to enhance their own skills and understanding in PHE. The relatively few trained who soon practiced what they learned within field implementation projects (IR 3) are serving to advance PHE and the PO. However, the numbers of both practitioners and field projects are too small to make a great difference. Leaders and champions who received some support from BALANCED are similarly too few in number and too little impacted to significantly contribute to the PO. 3. WHAT ARE THE KEY FACTORS CONTRIBUTING TO THE SUCCESSES AND SHORTCOMINGS OF THE PROJECT? BALANCED has nicely built on already existing environmental programs implemented by CRC and directed toward communities and environments in two settings: Tanzania and Ghana. Both sites promise some success for the project, as does the early award of a seed grant in Ethiopia and the ongoing implementation taking place in Zambia. Even in Tanzania, however, implementation within TCMP is less than it might have been. For one thing, TCMP has viewed PHE as an add-on, being implemented in a few relatively small sub￾areas, to what TCMP has been doing for some time. TCMP appears not to have embraced PHE as a strategic intervention that held promise for taking its conservation efforts to a new, higher level. This is shortsighted on the part of TCMP management itself as well as the oversight that CRC provides to this program. Strategically planning how PHE could have been inserted to enhance TCMP would have been a useful starting point. Instead, however, as with BALANCED more broadly, implementation of PHE activities simply began. Nor was the USAID Mission consulted early on as to how PHE could fit within USAID/Tanzania priorities. Despite close connections by virtue of the Pwani project, the ground was not prepared for a subsequent buy￾in that could have contributed to the broader success of BALANCED efforts in Tanzania. If early conversations took place with Juniper Neil, the USAID direct hire responsible for natural resource management at the Mission and a strong advocate for integration, these were not conveyed to the consultant for PFPI within BALANCED or to the line staff TCMP employee working for BALANCED. PHE inserted within TCMP has the elements of an integrated approach. Environmental work, including some involvement of communities and government in developing marine protected areas, provides the bedrock on which other interventions, such as micro-credit associations for livelihood development, have been built. Though sparse, a health infrastructure is in place, and a number of CBDs had already been trained (by GTZ) prior to BALANCED’s entry. Extensions of MID-TERM EVALUATION OF THE BALANCED PROJECT 57 the original environmental program, the latest as a result of a buy-in to conduct the Pwani Project, put in place a wide range of interventions, including for HIV/AIDS prevention. What BALANCED then did was place all of this within a broader context of inter-relations among population, health, and environment, and efforts to strengthen the family planning component. As such, however, and particularly because the project is not seen by TCMP as an enhancement to what it has already been doing for many years, this BALANCED field project is more layered than truly integrated. Furthermore, BALANCED has been implemented in a very small subsection of wards and villages covered by TCMP. Shortcomings are further exacerbated by the distance the project has maintained from mainstream development of family planning in Tanzania in recent years. After a crisis of contraceptive stock-outs and facing a government still reticent to change restrictive policies on service provision and contraceptive availability in Tanzania, family planning organizations, with leadership from USAID and the Ministry of Health, are moving to make up for lost time. The interim strategy is to make a fuller range of services and contraceptives more widely available by dispatching mobile service provision teams to areas throughout the country. This strategy is dependent on community mobilization of willing clients to connect with the arrival of mobile teams. Local ministry authorities request teams and CBDs are activated to ensure a ready clientele. Several USAID-supported family planning programs, among them EngenderHealth, Pathfinder, and PSI, play the intermediate role of providing resources to facilitate this by supporting government service providers as well as private outlets. Unfortunately, as a centrally funded project, BALANCED has not plugged into, nor was it even aware, that such developments were occurring and that they provide great opportunity for improving the use of family planning that its field implementation was promoting. A monthly coordination meeting among principal organizations, chaired by the ministry, is the primary mechanism for moving FP in Tanzania. BALANCED staff have not attended these meetings. Furthermore, neither BALANCED not TCMP were aware that mobile teams were being dispatched to meet FP service needs within both Pangani and Bagamoyo during early July, at the time of the evaluator’s visit. He learned of these plans during interviews with USAID and some of its key partners. The evaluator then asked TCMP and BALANCED staff if they knew of these occurrences. They did not. What was observed in Tanzania was symptomatic of what was reported by many other key informants as a major weakness of BALANCED—lack of strategic thinking or a strategic plan to guide implementation. Though clearly there are a number of structural barriers that the project has faced, such as being overly designed and under-funded, this lack of a guiding strategy was and is critical. Lack of an overall vision or guiding strategy within BALANCED has been further amplified by structural barriers of its own creation, the spatial and experiential distance separating the three partners, Particularly in these circumstances was it incumbent upon the project to create a unifying vision and strategy upon which to build the team that would implement this project. That some strategic elements are present in the CRC proposal makes these omissions during implementation even more glaring. 4. WHAT STEPS SHOULD USAID AND BALANCED TAKE TO ADDRESS THESE FACTORS (SHORTCOMINGS) IN THE LAST TWO YEARS OF THE BALANCED PROJECT? PLEASE IDENTIFY BOTH IMMEDIATE AND LONGER TERM STEPS. For Tanzania, some immediate steps arise directly from the evaluator’s findings during the country visit about family planning efforts. Local BALANCED staff should start to participate on a regular basis in the monthly meetings on family planning. While his presence would only represent a few small local areas, what the TCMP/BALANCED staff member could share about 58 MID-TERM EVALUATION OF THE BALANCED PROJECT the use of CBDs, peer educators, and cell phones at the community level would be widely useful in these broader efforts to coordinate provision of family planning. Furthermore such participation would provide up-to-date information on new funding streams from DFID and AUSAid, which are being programmed by USAID/Tanzania. This means of circumventing basket funding, which has dominated foreign assistance by all but USAID for several years, affords future opportunity to leverage funds for service provision improvement and expansion. Given that not much time remains in BALANCED, another Tanzania adjustment should also be given high priority. Rather than simply expand the model it has fielded within a few sites in Pangani District to a few more sites in Bagamoyo District, which is the current plan, BALANCED should consider revising its plans. A variant of the present PHE model, principally using peer educators, could be used to contiguously cover all or much more of the three districts that TCMP works in. BALANCED could concentrate on the approach already taken – strengthen family planning at the community level. BALANCED could then link local communities with mobile team services. The Mission may be receptive to such an approach, particularly if viable community-level work in Zanzibar and Pemba is included. One well-placed key informant suggested that, had BALANCED involved a partner organization such as Pathfinder in developing the concept note recently sent to USAID/Tanzania, Mission and family planning priorities for Tanzania would have been included to make a more compelling case for Mission buy-in. For the future, the evaluator suggests working with a key implementing partner for the Tanga Region, such as EngenderHealth, to formulate an approach to the Mission to extend family planning in Pangani, Bagamoyo, and Zanzibar. TCMP and BALANCED together have the experience to work sensitively and effectively in these new communities; USAID may well have the will and resources to support their doing so. The countries where BALANCED will operate are well set. Concentrating on activities in five countries—Ethiopia, Ghana, Philippines, Tanzania, and Zambia—may be as much as the project can reasonably do. It is not clear that the project has had enough impact to date or sufficient resources to merit continuance of technical assistance in two additional countries, Kenya and Uganda. If a well-structured seed grant emerges for funding in another country, perhaps that could be handled as well. However, BALANCED would do better to concentrate remaining resources on countries and programs where it already has traction. Given initial support provided in PNG, some additional work may continue. Given these considerations, BALANCED in consultation with USAID/PRH may want to revise the number of countries it has targeted for seed grants. BALANCED may also want to consider whether or not seed grants themselves are the key factor in determining countries where significant assistance will be provided, and adjust indicators and targets accordingly. Within these focus countries, BALANCED may better concentrate efforts in the time remaining to build capacity within and among those who are actually implementing PHE field projects. Beyond training, when and if needed, this would involve technical assistance for planning, problem-solving, and results. To an extent, such technical assistance is already provided to TCMP in Tanzania and more recently to AWS in Zambia. Periodic provision of technical assistance is also under way in Ghana, where it has been described as ―coaching in PHE‖. Efforts to provide similar levels of assistance to the Ethio Wetlands and Natural Resources Association (EWNRA) in Ethiopia have proven more challenging due to distance and difficulties in working with the implementing organization. (To its credit, BALANCED hopes to hold an end-of-seed￾grant meeting with EWNRA to learn from what has and has not gone well within this seed grant.) In addition to focusing on individuals active for PHE, some effort must also be given to fostering understanding and skills within the implementing organizations. To be clear, in the project’s remaining years, the recommendation is for training and technical assistance to produce PHE capabilities within individuals and organizations implementing PHE in field settings. Other activities, such as study tours to ongoing projects or workshops to exchange ideas MID-TERM EVALUATION OF THE BALANCED PROJECT 59 between those implementing PHE and those supporting other related programs, may be more useful approaches for attracting new PHE practitioners. This does not need to be limited to organizations that have received seed grants. By focusing on organizations already committed to implementation of PHE, BALANCED will promote sustainability for what it has helped to start. From the evaluator’s perspective, on-the-ground knowledge management can readily shift into the maintenance mode. The work on the two peer-reviewed journal articles as well as the four BALANCED tools must both be close to completion. The only remaining effort would largely be to ensure that those products are finalized and put on the website. It may be useful at this point in the project to unify remaining KM and website activities as the responsibility of one person within BALANCED. For the toolkit to be accessed and used more often may take time. However, what has transpired relatively quickly in Africa with the spread of cell phone use augurs well for the future of Internet connectivity. The website connection between K4Health PHE and the Ethiopia PHE Consortium is also worth monitoring for lessons it may yield in this regard. Despite evidence from the Tanzania country visit to the contrary, capacity building as articulated in the Year 4 Work Plan appears to be somewhat de-linked from field project implementation. Capacity-building activities for Year 4 will reach out to orient additional individuals and organizations at the Dakar Pre-Conference Workshop and the Coral Triangle Workshop. Such activities are justified as providing support for USAID’s global leadership in PHE. However, whether one-off workshops actually build capacity to implement PHE is a separate question. Should the Dakar workshop also heighten interest and opportunity for advancing PHE in other countries, such as Senegal and Gambia, where CRC has ongoing programs, it would have added value toward producing concrete results. That BALANCED will also provide some support for a refresher PHE design workshop in Ethiopia three years after the first workshop gives credence to the point of more closely linking capacity building with field implementation. Similarly, BALANCED’s attention to technical assistance and mentoring for pre- and post￾training support is conceptually, if not practically, attractive. However, the reality is that BALANCED is too thin on the ground to provide the amount of support needed. Follow up that has been provided has been mostly applied toward encouraging interest in PHE implementation, most likely requiring a seed grant at the minimum, along with extensive additional technical assistance. Particularly with the third IR consistently receiving the least amount of funding and with IR 1 considerably disconnected from field implementation, IR 3 activities have been invidiously disadvantaged within BALANCED. From the evaluator’s perspective, carefully considered strategic planning would have led to more emphasis on field implementation, closer linkages with capacity building, and more realistic assessments of what could be accomplished toward global leadership for PHE. Without innovative applications of PHE from which to learn, BALANCED is likely unable to provide much in the way of the global leadership that USAID had hoped for. CRC can be commended for the alacrity with which it has learned about and included PHE in two of its ongoing programs in Tanzania and Ghana. However, that CRC was also new to PHE may have meant that it was both fully accepting of the PFPI approach and less innovative about how this model could be more fully incorporated into ongoing environmental programs. Given then a relatively fixed geography within which to concentrate resources and effort, an appropriate endgame approach for BALANCED would be as broker rather than recipient for leveraged funds to improve and increase family planning. The intersection with the Pwani Project in Tanzania and its use of PEPFAR funding to integrate HIV/AIDS prevention into TCMP’s ongoing environmental program is a good model which BALANCED has built upon. BALANCED has already help to broker funding in Zambia with assistance provided to WCS in its application for USAID Flex Funds. In Ghana and Tanzania, TCMP and the Integrated Coastal 60 MID-TERM EVALUATION OF THE BALANCED PROJECT and Fisheries Governance (ICFG) Initiative provide mechanisms that could receive and use family planning funds beyond the life of BALANCED. Whether the same is also possible with EWNRA in Ethiopia would have to be explored. More broadly, there are other steps to consider in the near term. Instead of a management review centered on the Year 4 Work Plan, BALANCED and USAID/PRH may want to use this evaluation as a starting place for making changes and adjustments for the remaining years. This would entail digging into some of the details of how the project is currently organized and managed, and reassigning responsibilities as appropriate. BALANCED itself is well positioned to do so—staff know well what problems there are and will likely have good suggestions for resolving these. USAID’s role, particularly with the recent change of its Agreement Officer’s Technical Representative, could be reflective and accepting of what BALANCED offers as re￾adjustments. Within this context, it is not too late to organize the project along more strategic approaches. Ground work has already been done on this in the proposal itself and some subsequent efforts. Given the little time remaining, this strategy will have to be pragmatic as well as higher reaching. Inviting a core group of the best strategic thinkers and most involved managers to Rhode Island for a one-week session could well lead to productive and positive changes in how the project organizes for the last remaining years. An outside facilitator, experienced with management as well as strategic planning, could be a welcome addition to help BALANCED through this process. It may be painful, but it is likely to be highly productive. For the longer term, BALANCED would be well advised to seriously consider how it may live up to its acronym: Building Actors and Leaders for Advancing Community Excellence in Development. CRC indicated some interest in developing a short-term course for PHE leaders, and is well positioned to do so. As a culmination of BALANCED, such a program could leave a lasting legacy for the field: not only a cohort of energized and enhanced leaders for PHE, but also a course that can be sustained through self-financing in the years after BALANCED. This could go a long way to move this project toward global leadership for PHE. MID-TERM EVALUATION OF THE BALANCED PROJECT 61 ANNEX G. BALANCED PROJECT MIDTERM SELF￾ASSESSMENT 62 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 63 64 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 65 66 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 67 68 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 69 70 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 71 72 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 73 74 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 75 ANNEX H. BALANCED PROJECT INDICATORS AND PROGRESS TOWARD TARGETS June 30, 2011 Indicator Year 3 Total Year 3 Targets Cumulative Results to Date LOP Target Comments on Progress Toward LOP Target SO-1: Number of target organizations incorporating PHE tools, protocols, etc. into their work (PRH 1.1) 4 7 12 24 On track SO-2: Dollar value of funds leveraged from USAID Missions and non-USG sources to support PHE implementation and scale-ups (million dollars) (PRH 1.2) 12,890 600,000 1,639,982 2,500,00 On track 1.1 Number of individuals trained by BALANCED in PHE know-how and SOTA using 21st century learning tools 843 351 1193 200 Over target 1.2 Number of participants who received BALANCED training and/or mentoring that are now providing training or TA to others on PHE implementation 11 21 20 45 Under target 1.3 Number of new and/or strengthened partnerships established and actively involved in advancing and supporting wider use of PHE approaches as a result of BALANCED capacity￾building intervention (PRH 1.3) 9 4 11 16 On track 76 MID-TERM EVALUATION OF THE BALANCED PROJECT Indicator Year 3 Total Year 3 Targets Cumulative Results to Date LOP Target Comments on Progress Toward LOP Target 1.4a. Number of technical support interventions provided by BALANCED to Missions and to organizations to implement PHE within their programs (PRH 3.2) 25 24 50 30 Over target 1.4b. Number of new organizations receiving TA by BALANCED 1 12 19 30 On target 2.1 Tools, protocols, procedures, systems, methodologies, guides, curricula, or indices with demonstrated programmatic value developed or adapted for country and/or thematic contexts (PRH IR 2.1) 8 4 10 25 Under target— several tools are in final stages of completion 2.2a Number of success stories, peer review articles, conference papers, research studies documenting key actionable findings about the PHE approaches, their lessons extracted, and value-added (PRH IR 2.2) 21 12 38 12 Over target 2.2b. Number of peer reviewed articles and research studies 2 1 2 No target Over annual target 3.1 Number of PHE tools, methodologies, and actionable findings replicated in new countries and geographic areas (PRH 3.1) 9 4 16 9 Over target MID-TERM EVALUATION OF THE BALANCED PROJECT 77 Indicator Year 3 Total Year 3 Targets Cumulative Results to Date LOP Target Comments on Progress Toward LOP Target 3.2a Number PHE programs scaled up (PRH 3.4) 1 1 3 2 Over target 3.2b Number of geographical areas replicating PHE (PRH 3.4) 4 No target 5 6 On target 3.3 Number of BALANCED field site activities that produce results that can feed into KM activities (development of tools and success stories) 4 1 6 8 Over target 78 MID-TERM EVALUATION OF THE BALANCED PROJECT MID-TERM EVALUATION OF THE BALANCED PROJECT 79 ANNEX I. EVALUATION WORK PLAN BALANCED MID-TERM EVALUATION WORK PLAN (draft) Draft 10-16-09 Don Lauro Introduction: The BALANCED Mid-Term Evaluation Work Plan is presented in this document. On behalf of GHTech, Don Lauro as the Evaluator of the BALANCED Project is pleased to submit this Evaluation Work Plan as Deliverable #1 under Scope of Work for the GR/PRH: The BALANCED Project Performance Evaluation. This Work Plan has been developed in accordance with the SOW and guidance subsequently provided in June 13th and 16th meetings with Heather D’Agnes, Lead Technical Advisor for PHE and AOTR for the BALANCED Project. As agreed in those meetings, the preliminary outline of the draft report and memo will not be submitted until the evaluator returns from Tanzania, on or about July 11th. The Work Plan is formulated within six items as follows: 1. Proposed Methodology for the Evaluation 2. Evaluation timeline 3. Questions for key informant interviews 4. List of organizations/individuals being interviewed (in DC, at CRC, and elsewhere) 5. List of documents provided to the evaluator by USAID to complete and execute the SOW 6. Draft team schedule for country visit to Tanzania and criteria for in-country site visit selection ITEM 1: PROPOSED METHODOLOGY FOR THE EVALUATION a) Document Review: The BALANCED Evaluation began with a review of a comprehensive set of BALANCED and related population-health-environment (PHE) documents. These were provided by USAID at the start of the assignment, and included a number of internal project documents, publicly available reports, and selected web sites. Supplementary documents undergoing review also include published papers; program reports; unpublished technical papers, reports, data compilations, and presentations; and trip reports. Throughout the key informant interviews, the site visit to the Coastal Resource Center (CRC) at the University of Rhode Island, and the country visit to Tanzania, the evaluator will also collect and review additional documentation produced by or pertinent to BALANCED, implementing partner organizations, and others working in the field of population, health, and/or environment. A list of all documents reviewed and found pertinent to this evaluation will be included as an appendix of the Evaluation Report. b) Interview key informants: A key informant is defined as someone who has breadth as well as depth of knowledge about this particular project and/or the broader field of PHE. While most key informants will be within USAID, the BALANCED Project, and partner implementing organizations, some may be unconnected to the project but have broad perspectives to offer on PHE. During the week of June 13-17, the Evaluator conducted on-site, in-person interviews with cognizant USAID/W staff, including the AOTR, BALANCED staff at CRC (both in Washington DC and the University of Rhode Island), and relevant representatives of partner organizations in 80 MID-TERM EVALUATION OF THE BALANCED PROJECT the Washington DC metropolitan area. During the following week, (June 20-24), the evaluator will also interview (remotely) key informants of other partner organizations, other stakeholders, other experts knowledgeable about PHE, particularly advisory committee members. Some additional and follow-up interviews will likely also occur after the evaluator returns from the country visit. An initial list of key informants and organizations has been provided by USAID/W BALANCED Project management. Additions to this initial list occurred during key informant interviews, particularly with USAID and BALANCED staff. (For the complete list, see Item 2). During the course of this evaluation, the Evaluator may further build on this list and formally interview, in person or by phone, others relevant to this Evaluation. This list will be updated periodically, with a final list appended to the Evaluation Report. To collect relevant data from key informants, the Evaluator will conduct semi-structured interviews using open-ended questions. Based on the SOW, background documents, and guidance provided by USAID/W, for each interview or set of interviewees, key questions will be extracted from a longer list of questions developed in relation to the seven questions contained in the SOW. The generic question list for the key informant interviews may be found in Item 3 of this Work Plan. In order to elicit information relevant to completing this SOW, the evaluator as appropriate for particular key informants also asks other questions that may not be on this list. In-person interviews of key informants are preferred. For this reason, a number of key informant interviews occurred during the week of June 13th to 17th both in DC and Rhode Island, and a number of others will occur in Tanzania. However, remote phone or email interviews will be used to access other key informants, including those working outside of DC and RI, or those involved with PHE in countries other than Tanzania. Gathering information from some key informants may require more than one interview. This will clearly be the situation with the USAID AOTR and also likely for some core BALANCED staff. This will enable the evaluator to concentrate on and dig deeper into various objectives and questions in the SOW, particularly as data collection and analysis proceeds. Some re-interviews with selected informants may occur after the Evaluator returns from the Tanzania country visit. c) Site visits: The Evaluator will travel to Tanzania for the purpose of conducting visits to BALANCED project implementation sites and interviewing key informants. Sites visited will include communities within the Pangani and Bagamoyo regions, and possibly field sites where partner organizations are implementing related activities. Key informants to be interviewed will include relevant USAID personnel; in-country BALANCED staff; other international and local NGOs; and government counterparts. A preliminary draft schedule of the two-week visit to Tanzania is attached as Item 6. Site visits will provide some opportunity for the Evaluator to collect a different level of data than that yielded by interviews. However, opportunities to do so may be limited by time and circumstance. Particularly in Tanzania, however, the evaluator would hope at times to observe project activities in action. Because the country visit is but one part of an ongoing evaluation, the Evaluator does not anticipate a formal debriefing on findings with USAID/Tanzania. However, as useful and desired by Mission, the Evaluator would be pleased to informally provide initial impressions based on time available and information gathered during the country visit. d) Data Analysis Plan: Questions for the interviews of key informants will be drawn from the question list developed in relationship to the seven main questions presented in the SOW. MID-TERM EVALUATION OF THE BALANCED PROJECT 81 Responses to interview questions will be concurrently recorded (written notes) and subsequently transferred into computer files. To analyze responses by key informants the evaluator will both comprehensively review responses and conduct content searches for key terms and concepts. Based on this, trends and themes will be identified as related to each particular question posed in the scope of work. Preliminary summations in terms of SOW questions will be checked against information from project documents and other sources, including what was learned during site visits in the field. Triangulation will occur in terms of what documents indicate BALANCED was meant and/or did accomplish, what key informants reported about BALANCED, and what the Evaluator learned from the combination of document review, interviews, and site visits. ITEM 2: EVALUATION TIMELINE Month—Dates Tasks June 6-8 Preparation for assignment: Review of background materials (SOW, BALANCED website, and related materials (e.g., BALANCED Reports, Work Plans, Semi-Annual Reports, etc. June 9 Begin Scheduling Interviews and Travel to DC June 10 Scheduling Interviews; and begin developing questions for key informant interviews based on SOW questions; administrative meeting with GHTech. June 13 Initial meeting with USAID/W and complete draft of key informant interview questionnaire. June 13-16 BALANCED presentation of Self-Assessment PowerPoint and DC-based interviews with key informants at USAID/W; BALANCED; and partner organizations as per list of key informants provided by USAID June 17 Continue Interviews at University of Rhode Island with CRC staff who have project responsibilities within BALANCED. June 18 Travel to Davis, CA. June 19-23 Key Informant Interviews continue remotely with some additional key informants in DC and others located elsewhere; complete draft of BALANCED Evaluation Work Plan and send to GHTech and USAID June 23-24 Travel to Tanzania June 25-July 10 Country visit to Tanzania for project site visits and interviews with key informants within BALANCED and partner organizations July 10-11 Return Travel to Davis, California July 12 –15 Analysis and synthesis of data/information; check in with USAID/W; conduct additional key informant interviews and follow-up interviews as needed. July 15 - 24 Draft Report July 25 Submit Draft Report July 26 Evaluator returns to DC for debriefing on findings and recommendations July 27-29 Debrief Meetings in DC August 1 Evaluator finalizes draft and submits to GHTech 82 MID-TERM EVALUATION OF THE BALANCED PROJECT ITEM3: KEY INFORMANT QUESTIONNAIRE KEY INFORMANT INTERVIEW QUESTIONS BALANCED EVALUATION Legend: SOW Questions Italicized Interview Questions In Bold BACKGROUND QUESTIONS: B1 How did you come to this point in your career, significant responsibilities over/within … (e.g., a project that seeks to advance PHE)? BALANCED GENERAL QUESTIONS: BG1 Please describe a typical work day for you. a. What kinds of things do you do on a fairly regular basis, including those apart from BALANCED? b. What kinds of things do you do on a fairly regular basis in relation to BALANCED, and specifically IR1, IR2, and/or IR3? BG-2 What in your previous professional experiences most prepared you for implementing this project BG-3 What advantages and disadvantages have their been as a result of the main implementing partner for a USAID cooperative agreement being a university. BG-4 How has the project dealt with the complexity and built-in challenges of key staff being both part time and based in different locations and time zones. BALANCED INTERMEDIATE RESULTS QUESTIONS: 1. To what extent has the BALANCED project achieved the expected results under each intermediate result: – IR 1: Capacity built for integrated PHE implementation (for expected results, refer to pages 9- 12 of the Technical Application (TA), originally labeled IR 2; also see the Performance Management and Reporting Plan (PMP) for descriptions of indicators and targets). MID-TERM EVALUATION OF THE BALANCED PROJECT 83 INTERMEDIATE RESULT 1: BUILD CAPACITY 1:BIR1-1 BALANCED seeks to strengthen PHE capacities … a. How does it do this? b. What has its impact been to date? c. What challenges has it faced? 1:BIR1-2 Describe and weight the mix through which BALANCED seeks to strengthen and institutionalize local capacities for implementing PHE field projects. a. What has its impact been to date? b. What challenges has it faced? 1:BIR3-3 Have there been changes in PHE capacity within countries, such as Ethiopia, Tanzania, and the Philippines, since October, 2008 when BALANCED got under way? a. To what extent can any change be directly attributed to BALANCED and how did this come about? b. What was its contribution? 1:BIR1-4 Part of BALANCED places emphasis on developing and supporting champions for PHE. a. To what extent can the emergence of PHE champions be directly attributed to BALANCED and how did this come about? b. What were BALANCED’s specific contributions? c. Should/could BALANCED have done more to create and support PHE champions? 1:BIR1-5 During the last twelve months how often and for what purposes have you traveled from the office to provide technical assistance or support of some kind? a. Focusing on what you would consider the most successful occasion when you worked in the field, please provide details. b. Concentrating on an assignment to the field that was not optimally successful, please analyze what in retrospect could have been done differently? INTERMEDIATE RESULT 2: KNOWLEDGE MANAGEMENT IR 2: PHE knowledge and tools developed, organized, synthesized and shared (refer to pages 6-9 of the TA for expected results, originally labeled IR 1; also see the PMP for descriptions of indicators and targets). 84 MID-TERM EVALUATION OF THE BALANCED PROJECT 1:BIR2-1 What readings or scholars are you familiar with who have best informed and stimulated your thinking about how population, health and environment may be interlinked or interactive? Should economic enterprise or livelihoods not also be a part of the interactions that occur and are interlinked at the community level? 1:BIR2-2 What do you expect BALANCED to provide in terms of global leadership, knowledge building, and collaboration? a. What are the global leadership strengths to date? b. What are the global leadership weaknesses or shortcomings and how may these be improved? 1:BIR2-2 At the global and/or regional level, to what extent and how has BALANCED helped advance PHE, supported advocacy efforts, and developed other kinds of collaborations with partners? a. With which organizations has collaboration been most effective and/or groundbreaking? b. Have there been missed opportunities for collaboration or areas where more needs to be done? c. In what ways has BALANCED been pro-active in helping global partners strengthen their own PHE programs? 1:BIR2-3 Cite specific examples of best practices, useful approaches, and/or tools drawn from previous experiences within PHE that have been adapted by BALANCED. a. Which do you identify as the most useful and likely to be most widely used? Where have these been described? b. Have these actually been applied within field activities directly supported by BALANCED? c. What other best practices is BALANCED learning about that it intends to apply? 1:BIR2-5 Do you know of any tools or toolkits that you or others have found helpful for understanding how to develop and implement integrated PHE field projects? If so, who developed these, why are they useful, and where can they be found? BIR-3 How often and for what purposes do you yourself use the BALANCED website? a. In addition to the use-data about the web site, what other information (anecdotal or otherwise) do you have that the BALANCED website is used and useful for practitioners of PHE? b. What more is planned to increase relevance and usage of this website or other means to reach a wide technical audience? MID-TERM EVALUATION OF THE BALANCED PROJECT 85 INTERMEDITATE RESULT 3: FIELD PROJECTS IR 3: Results-oriented PHE field activities implemented in areas of high biodiversity (refer to pages 12-14 of the TA for expected results; also see the PMP for descriptions of indicators and targets). 1:BIR3-1 Another of the BALANCED Project components focuses on implementing PHE projects on the ground. In addition to providing ongoing support to IPOPCORM in the Philippines, the project also supports coastal and uplands projects in Tanzania and Ethiopia. What do you know about BALANCED successes or shortcomings at this level? 1:BIR3-2 What do you expect BALANCED to provide in terms of country-level field projects? a. What have been some outstanding successes to date? b. Where have there been significant shortcomings or failures? c. What lessons have been learned? 1:BIR3-3 Within countries where BALANCED operates, has the terrain been mapped for ongoing and/or potential PHE activities? 1:BIR3-4 With which countries has BALANCED enjoyed the most success during the first three years? a. Which in-country partners have achieved the greatest successes or have the most potential? b. Have some anticipated or unanticipated country-level partners experienced an expanded or diminished role during the course of BALANCED? c. Where and with which organizations do you wish that BALANCED had been able to do more? 2. By implementing activities under IR 1, IR 2, and IR 3, to what extent has the BALANCED Project been effective in achieving the Project Objective: “Advance and support wider use of effective PHE approaches”? Evidence of achievement of the Project Objective would include, but is not limited to: – Effective PHE approaches are fine-tuned and readily available to donors and implementing agencies. – Increase in the number of organizations adopting and implementing PHE approaches. – A cadre of experienced PHE professionals, including from developing countries, is available to implement PHE programs and provide technical support to new and continuing PHE efforts. – Increased support (financial or otherwise) for PHE approaches beyond USAID. 2POIR-1 What specific tools, activities, and/or approaches has BALANCED used with regularity and success to advance PHE? 2POIR-2 86 MID-TERM EVALUATION OF THE BALANCED PROJECT Are there other activities and approaches that BALANCED could have usefully invested in to support champions and leaders for PHE? 2POIR-3 What have been the successes and shortcomings around leveraging funds or in-kind contributions for PHE,from USAID/Missions, other USAID partners, other NGOs, and other funders? What could have been done differently to improve things? 2POIR-4 Would BALANCED or PHE have advanced further if the project had done more earlier with field implementation rather than managing knowledge? 2POIR-5 Overall, has BALANCED been responsive to the field’s technical needs? a. What is the skills mix within BALANCED staff and is it sufficient to respond to field needs for technical and managerial support? b. In what areas can BALANCED improve its response to field needs? c. Does BALANCED have more requests for TA than it can fulfill? In what areas? 2POIR-6 What types of technical inputs does BALANCED provide to other USAID projects and organizations with which it collaborates? a. Please provide examples of specific inputs provided by BALANCED and assess their relevance and technical soundness. b. Are there also some examples where technical assistance has fallen short of increasing the capacity of an organization? c. Are there examples of collaborations that did not develop or continue because what BALANCED could provide did not or was not perceived to match what was needed? Please elaborate. 2POIR-7 What opportunities are there to advocate for and leverage funding from different sources and at different levels? a. Give examples of successes increasing in-country support to advance PHE. b. Give examples and opportunities for increasing support for PHE. 2POIR-8 Please describe your role and/or your understanding of how the PPM indicators were developed? a. How would you describe the indicators that are now tracked: input, output, outcome, or other? b. What other indicators could one envision as important to track over the long term to measure achievements of a project like BALANCED? MID-TERM EVALUATION OF THE BALANCED PROJECT 87 3. What are the key factors contributing to the successes and shortcomings of the project? 3:POIR-1 What from your point of view has been the most effective part of BALANCED? What is the least effective? 3:POIR-2 How does BALANCED currently build capacity within organizations—through subcontracts, technical assistance, training, staff in residence, other approaches? Please provide examples where this has proceeded well and not so well. 3:POIR-3 Are there examples where knowledge management and field implementation of PHE has not yielded results hoped for? 3:POIR-4 What are BALANCED’s unique contributions or special capabilities for advancing PHE? 3:POIR-5 Are there near-term future opportunities for leveraging funds and/or advancing PHE? 3:POIR-6 What contributions/expectations does your part of the management structure or project have to make to supporting BALANCED and how do you go about doing this? 3:POIR-7 What are the high and low points of working with or within BALANCED since this Project started, in late 2008? Please provide concrete examples of each? Were lessons learned from these experiences, and (if so) how were they shared with others? 3:POIR-8 What during the last year or over the course of BALANCED are you personally most proud of having done with or within BALANCED? What do you have regrets about having done or not done that would have contributed to BALANCED? 3:POIR-9 What technical and managerial support does BALANCED receive on a regular basis from USAID? Does BALANCED need more or less of this support? a. Please give specific examples of where support provided by USAID has been most helpful to achieving the BALANCED project objective and IRs. b. Where has USAID’s assistance and support fallen short or worked against BALANCED being effective? 88 MID-TERM EVALUATION OF THE BALANCED PROJECT 4. What steps should USAID and BALANCED take to address these factors (shortcomings) in the last two years of the BALANCED project? Please identify both immediate and longer term steps. 4:BF-1 In the remaining two years, what would you like to change either about the project or about your own work? 4:BF-2 At this midpoint in implementing BALANCED, are there any significant adjustments that you would like to consider? Are there adjustments within the current management structure or BALANCED approaches that would be helpful? 4:BF-3 What are the most useful and fulfilling parts of your work? What makes you frustrated or unsatisfied? Do you have suggestions for improving the Project, what are these? 4:BF-4 In your view, is the BALANCED project objective the right one and are the IRs rightly ordered and weighted (funded)? Are the skills among staff the right ones to achieve these? BALANCED FUTURE QUESTIONS: 1. Are there activities under BALANCED that are not contributing as expected to the achievement of the Project Objective? How could these activities be adjusted to have more impact? BF1-1 What challenges and contradictions are there in fielding an approach that embraces ―advance and support wider use of effective PHE approaches‖? How does BALANCED balance such long￾term ambitions with the relatively small amount of resources in this project? 2. What are activities, not included in BALANCED, that should be added to the follow￾on project in order to achieve the Project Objective? BF2-1 What activities and/or approaches, not included in BALANCED, should be added to the follow￾on project in order to achieve the Project Objective, ―Advance and support wider use of effective PHE approaches‖? BF2-2 In your experience, what are the best PHE field projects that you have seen or have supported? What elements within these projects or what aspects of their implementation account for their high standard of excellence? In other words, what are some of the secret ingredients to success? MID-TERM EVALUATION OF THE BALANCED PROJECT 89 3. Should the Project Objective be changed for the follow-on? If so, how? BF3-1 If the project were starting anew, what would you change, structurally as well as substantively? BF3-2 If you were the director of BALANCED, or of USAID, what would you change about this project, USAID’s approach to PHE, or USAID itself? BF3-3 Much of the knowledge management seems oriented towards accumulating and disseminating practical how-to knowledge about implementing PHE projects. Is or should there also be effort to build knowledge by investing in theory or understanding about how population, environment, and health are linked? BF3-4 How do you envision PHE 10 years from now in the level and type of countries where BALANCED now operates? BF3-5 What question that I haven’t asked of you, would you wish that I had asked? And what is your response or responses to that/those question(s)? TANZANIA SPECIFIC (PRELIMINARY QUESTIONS): TAN-1 Please provide a brief narrative history of conservation efforts, health programs, as well family planning and contraceptive use in Tanzania. a. What have been the ups and downs, success and failures in advancing conservation, health, and family planning? b. What is the current status of PHE or integrated approaches at the community level in Tanzania? TAN-2 Would you provide examples of how often and for what purposes you have been in contact with the BALANCED staff? a. Please describe the situation that occurred when contact was made with either BALANCED and/or the USAID mission regarding PHE. b. Was any further assistance from BALANCED provided and what have been the results of that? 90 MID-TERM EVALUATION OF THE BALANCED PROJECT TAN-3 How and with what effectiveness has the BALANCED Project worked in Tanzania over the last three years? a. What has the USAID BALANCED Project provided to directly advance PHE programs in Tanzania? b. Outside of direct financial or technical support to Pangani, what other support has BALANCED provided during the last three years? c. Has Balanced advanced, hindered, or been neutral with regard to family planning progress in the country? TAN-4 Have you HAD technical assistance from BALANCED? How often and to do what? What, if any, lasting impact has this assistance had on your organization and its ability to implement PHE? Give specific examples of when this has worked well and when this hasn’t. ITEM 4: LIST OF ORGANIZATIONS/INDIVIDUALS INTERVIEWED (IN DC, AT CRC, AND ELSEWHERE) WASHINGTON, D.C. USAID—Office of Population and Reproductive Health Ellen Starbird, Deputy Director Elizabeth Schoenecker, Chief, PEC Heather d’Agnes, Technical Advisor for PHE and BALANCED Agreement Officer’s Technical Representative BALANCED/D.C Linda Bruce (CRC), Director Janet Edmond (CI), Deputy Director Team Presentation of Self Assessment—Linda Bruce; Janet Edmond, and (by phone) Leslie Squillante Wilson Center Environment Change and Security Program (ECSP) Geoff Debelko, Director, ECSP Meaghan Parker, Writer/Editor, ECSP World Wildlife Fund Cara Honzak, PHE Technical Advisor Population Action International Roger-Mark deSouza, Deputy Director for Research Population Reference Bureau Jason Bremner, Program Director, Population, Health, and Environment (phone interview on June 22nd) Jane Goodall Institute Alice Macharia, Director for East Africa Programs (phone interview on June 22nd) MID-TERM EVALUATION OF THE BALANCED PROJECT 91 Johns Hopkins/CCP Guy Chalk, IT Manager, Knowledge for Health Project (by email) BALANCED Project, Coastal Resource Center, University of Rhode Island: Brian Crawford, Director International Programs (by phone) Leslie Squillante, Deputy Director Cindy Moreau, Financial Administration Elin Torrel, Don Robadue Bob Bowen Experts, Partners, and Donors outside DC: Pathfinder International: Sono Aibe, Senior West Coast Representative for Strategic Initiatives Packard Foundation: Sahlu Haile, Senior Representative for sub-Saharan Africa Yemeserach Belaynay, Foundation Representative for Ethiopia Bernd Cordes, Foundation Program Officer, Conservation Program Environmental Leadership Program, University of California Berkeley Robin Marsh, Co-Director Nagash ELP 2011 Participant—Ethiopia Patrick ELP 2011 Participant—Tanzania Evaluation and Research Technologies for Health (EARTH) Inc. Lynn Gaffikin, President (and Advisory Group for BALANCED) John Snow, Inc Nancy Harris, Vice President (and Advisory Group for BALANCED) Other Partners and Experts in Tanzania and Elsewhere (TBD): ITEM 5: DOCUMENTS THAT THE TEAM HAS RECEIVED (FROM USAID/W AND COLLECTED TO COMPLETE AND EXECUTE THE WORK PLAN: Prior to arrival in Washington, DC, the Evaluator received a comprehensive set of documents from USAID/W on BALANCED. These documents include the BALANCED RFA, BALANCED Work Plans and Management Reports, semi-annual reports, and semi-annual reports, publications list. The evaluator has also collected during the initial phase of this evaluation a number of supplementary materials, with the most relevant and useful included in the list below. This list of documents that form part of this evaluation will be updated and listed in an Appendix of the report. List of background documents for BALANCED Evaluation BALANCED Project planning and management documents PHE Technical Leadership Cooperative Agreement RFA 92 MID-TERM EVALUATION OF THE BALANCED PROJECT BALANCED Technical Application in response to the RFA Performance and Management and Reporting Plan Communications Strategy Resources Mobilization Strategy Annual Work plan, Year 1 Semi-Annual Report, #1 Semi-Annual Report, #2 Performance Management Review Report of the BALANCED project (2009) Annual Work plan, Year 2 Semi-Annual Report, #3 Annual Work plan, Year 3 Semi-Annual Report, #4 Advisory Committee notes, 2009 Take Home Messages from Advisory Committee Meeting Results Review - Year 2 Performance Management Review Report of the BALANCED project (2010) Semi-Annual Report, #5 BALANCED Status evaluation chart, indicator summary report BALANCED Project deliverables and contacts List of BALANCED key collaborating partners (compiled by BALANCED staff) List of other stakeholders, partners List of BALANCED activities by country List of all products (training curricula, websites, documents etc.) produced by BALANCED and weblinks to the products. Electronic copies will be provided if weblinks are unavailable. Including link to PHE Toolkit (www.k4health.org/phe). PHE South-to-South Exchange Participants’ Directory USAID background information PRH Results Framework PRH Results Framework with indicators Review of Population, Health, Environment Programs supported by the Packard Foundation and USAID (2005) Assessment of USAID’s Population and Environment Projects and Programming Options (2007) MID-TERM EVALUATION OF THE BALANCED PROJECT 93 Additional Documents of Interest Linking Population, Health, and the Environment, in Mount Sinai Journal 78:394-405, 2011 BALANCED Newsletter, June 2011 Healthy People, Healthy Ecosystems: WWF, USAID, Johnson and Johnson (2008) Healthy Families, Healthy Forests, USAID and Conservation International BALANCED Newsletter: PHE Approaches, April 2010 The Importance of Population for Climate Change, PAI Fact Sheet #37 Sharing the Forest: Protecting Gorillas and Helping Families in Uganda; Focus on Population, Environment, and Security; ECSP Wilson Center, Oct. 2008 ITEM 6: DRAFT SCHEDULE FOR COUNTRY VISIT TO TANZANIA Tentative Schedule for USAID Mid-term evaluator for BALANCED Project (Revised 6.21.11) June 23ish—Ricky arrives and will be staying at the Markum Hotel June 25—Don Lauro arrives TCMP to pick-up from the airport—Juma, please confirm this with Don. Don will be staying at the Seacliff hotel. June 26 to June 28—Don is scheduling his own meetings. If assistance is needed, Don will communicate directly with Juma. June 29 to July 2 - Visit Pangani and Bagamoyo sites Juma is arranging the field visit (and Esther the lodging) and will send the schedule of the field visit to Don—with a copy to Linda. Ricky and Juma will accompany Don on the field visit. July 3 to 4—visit Arusha (AWF) - this includes travel and talk time Don will communicate with Juma about the visit and arrangements. Ricky will accompany Don to Arusha, if Don chooses. To be discussed between Ricky and Don. See AWF contact information below. July 5-6 visit Iringa (WCS) —this includes travel and talk time Ricky will contact Bkari when he is Tanzania and inquire if a meeting is possible. If not, we just let it go. If a visit is made to WCS, Don will inform Ricky whether he would like Ricky to accompany him or not. See WCS contact information below. July 7—visit TCMP in Bagamoyo Juma will arrange a pick up to take you to TCMP’s office in Bagamoyo to meet with Daffa and other staff. 94 MID-TERM EVALUATION OF THE BALANCED PROJECT For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com