MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT (CONTRACT NUMBER: FAO-C-00-98-00079-00) By: David F. Pyle Garth Osborn Maria Lourdes Francisco Submitted by: LTG Associates, Inc. and TvT Associates, Inc. Submitted to: The United States Agency for International Development Under Contract No. HRN-I-00-99-00002-00 April 2001 Midterm Evaluation of the Child Survival Technical Support Contract was prepared under the auspices of the U.S. Agency for International Development (USAID) under the terms of the Monitoring, Evaluation and Design Support (MEDS) project, Contract No. HRN-I-00-99-00002-00. The opinions expressed herein are those of the authors and do not necessarily reflect the views of LTG Associates, TvT Associates, or USAID. Information about this and other MEDS publications may be obtained from: Monitoring, Evaluation and Design Support (MEDS) Project 1101 Vermont Avenue, N.W., Suite 900 Washington, DC 20005 Phone: (202) 898-0980 Fax: (202) 898-9397 scallier@ltgassociates.com ACKNOWLEDGMENTS The evaluation team greatly appreciates the time and thought that the private voluntary organizations (PVOs) contributed to this evaluation. Without exception, they responded to the team’s telephone interviews seriously and honestly. We also want to thank those PVOs that responded to the questionnaire. This exercise could not have been completed without their cooperation. We also thank those at the United States Agency for International Development (USAID), especially the Office of Private and Voluntary Cooperation (PVC), and the Child Survival Collaboration and Resources Group (CORE) who participated in the evaluation and shared their experience and vision. The Monitoring, Evaluation and Design Support (MEDS) project supported the evaluation team throughout its mission. Special thanks go to the project director who facilitated the team planning meeting, which was helpful in clarifying the roles and responsibilities of the team in carrying out the evaluation. The myriad of logistic details were handled efficiently and calmly by the task order coordinator. ACRONYMS AND FOREIGN TERMS BASICS Basic Support for Institutionalizing Child Survival BHR Bureau for Humanitarian Response CA Cooperating agency CORE Child Survival Collaboration and Resources Group CS Child Survival Division CSAP Child Survival Application Program CSGP Child Survival Grant Program CSSP Child Survival Support Project CSTS Child Survival Technical Support CTO Cognizant technical officer DHS Demographic and Health Survey DIP Detailed Implementation Plan DOSA Discussion-oriented self-assessment EHP Environmental Health Project FANta Food and Nutrition Technical Assistance GEM Global Excellence in Management G/PHN Bureau for Global Programs, Field Support and Research Center for Population, Health and Nutrition HFA Health Facility Assessment HIV/AIDS Human immunodeficiency virus/acquired immune deficiency syndrome HMIS Health management information system IMCI Integrated management of childhood illness ISA Institutional Strength Assessment JHU Johns Hopkins University KPC Knowledge, practice, and coverage LQAS Lot Quality Assurance Sampling MAPS Minimal acceptable performance standards MEDS Monitoring, Evaluation and Design Support MOU Memorandum of Understanding MOBIS Management Organization and Business Improvement Services NGO Nongovernmental organization OCA Organizational Capacity Assessment OP Office of Procurement ORT Oral rehydration therapy PAHO Pan American Health Organization PDF Portable document format PPEM Participatory Program Evaluation Methodology PVC Office of Private and Voluntary Cooperation PVO Private voluntary organization QAP Quality Assurance Project R4 Results Review and Resource Request RPM Rational Pharmaceutical Management SO Strategic Objective SOTA State of the art SOW Scope of work STI Sexually transmitted infection TRM Technical reference materials UNICEF United Nations Children’s Fund USAID United States Agency for International Development CONTENTS ACKNOWLEDGMENTS ACRONYMS AND FOREIGN TERMS EXECUTIVE SUMMARY .............................................................................................. i I. INTRODUCTION.................................................................................................. 1 A. Purpose.......................................................................................................... 1 B. Evaluation Team............................................................................................ 1 C. Methodology ................................................................................................. 1 II. BACKGROUND.................................................................................................... 5 A. Child Survival Grant Program........................................................................ 5 B. Technical Support Contracts.......................................................................... 6 III. TECHNICAL FINDINGS AND RECOMMENDATIONS..................................... 9 A. Summary of Findings..................................................................................... 9 B. Contract Results: Findings and Recommendations......................................... 9 C. Crosscutting Issues and Themes................................................................... 21 Utilization of Technical Assistance ........................................................ 21 Field Orientation.................................................................................... 26 Training/Capacity Building in Core Capabilities.................................... 27 Materials and Dissemination .................................................................. 31 Linkages with Cooperating Agencies..................................................... 34 Strategic Planning .................................................................................. 36 IV. FINDINGS ABOUT PROJECT ADMINISTRATION ......................................... 39 A. CSTS Startup............................................................................................... 39 B. USAID Management ................................................................................... 40 C. Broader PVC Needs..................................................................................... 41 V. FINDINGS REGARDING PROJECT BUDGETING AND FINANCING ........... 43 A. Fiscal Responsibility.................................................................................... 43 B. Project Expenditures.................................................................................... 44 C. Budgeted, Expended, and Projected Costs Analysis..................................... 45 VI. CONCLUSIONS.................................................................................................. 47 ANNEXES A. Scope of Work ........................................................................................................ 51 B. Persons Contacted ................................................................................................... 57 C. Crosscutting Questions............................................................................................ 63 D. PVO Questionnaire ................................................................................................. 67 E. Miniquestionnaire ................................................................................................... 71 F. CSTS’ Publications and Trainings........................................................................... 73 G. References............................................................................................................... 75 H. BHR/PVC—CSTS Performance Work Statement.................................................... 81 I. Results, Indicators, and Performance Standards....................................................... 87 J. Evaluation Schedule ................................................................................................ 99 K. Current Child Survival PVO List........................................................................... 101 FIGURES Figure 1: Results Per Contract..................................................................................... 10 Figure 2: Results Per Reality....................................................................................... 12 Figure 3: Frequency of Use: New Versus Old ............................................................. 24 Figure 4: Frequency of Use: Far Versus Near.............................................................. 24 Figure 5: Frequency of Use: Small Versus Large ........................................................ 25 Figure 6: Project Expenditure by Category.................................................................. 44 i EXECUTIVE SUMMARY The Child Survival Technical Support (CSTS) project is the technical assistance contract to support the private voluntary organizations (PVOs) implementing child survival projects funded by the United States Agency for International Development (USAID), Bureau for Humanitarian Response (BHR), Office of Private and Voluntary Cooperation (PVC). This midterm evaluation of the performance-based CSTS contract was carried out to assess its performance for BHR/PVC and to advise USAID on whether to continue funding for the last two years of the contract. If contract continuance was recommended, suggestions on improving the effectiveness of the project were to be included. At the same time, the team was charged with reviewing the contract performance standards and suggesting modifications. The evaluation team utilized a methodology consisting of several elements. Of primary importance were the interviews of a large number of people who are or have been involved in the Child Survival Grant Program (CSGP) during the two and a half years the current contractor (MACRO, Inc.) has managed the CSTS contract. The team interviewed 56 individuals, including persons from: ß BHR/PVC (9), ß the Bureau for Global Programs, Field Support and Research Population, Health and Nutrition (G/PHN) (3), ß the Child Survival Collaboration and Resources Group (CORE) board members and working groups (5), ß MACRO CSTS staff (9), ß cooperating agencies (CAs) (4), ß Johns Hopkins University (1), and ß PVOs (23). Two consultants to the project were also interviewed. To maximize feedback and encourage as broad a view as possible, the PVOs were divided into four groups by ß size (2 small and 2 large), ß length of time in the CSGP (2 new and 2 old), ß location (2 far and 2 close), and MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT ii ß those not currently in the CSGP (2). While most of those interviewed were based at the headquarters office, some field-based staff were interviewed by telephone. In addition, the evaluation team developed and disseminated a miniquestionnaire to give all CSGP grantees (especially those not interviewed) an opportunity to describe their experience with the CSTS project and their ideas regarding its future. Moreover, the evaluators reviewed a large volume of project documents. Background The CSGP was begun in the mid–1980s with the beginning of the Congressionally earmarked child survival funds and as part of USAID's effort to reduce child mortality. Since its inception, the program has changed in terms of: ß interventions (grown from two to nine), ß data collection tools (from an almost exclusive reliance on knowledge, practice and coverage [KPC] to the use of a variety of instruments), ß length of grant (increased from three to five years), and ß amount of funding per agreement (from an average of $500,000 to $1.3 million). Simultaneously, the environment in which the child survival projects are implemented changed significantly, especially in the latter part of the 1990s, when the Internet became a reality and CORE, an association of PVOs involved in child survival programming, was formed and became an active force. The first two Child Survival Support Projects (CSSPs) focused their attention on technical interventions and the development and institutionalization of the KPC. However, PVC wanted to concentrate on building the PVOs’ capacity in organizational and program management as part of USAID’s reengineering effort and the increased focus on results. This shift gave rise to the CSTS contract. At present, 27 U.S.–based PVOs are receiving child survival grants to implement 79 projects in 36 countries. After a difficult first year that featured a steep learning curve for all concerned, the contractor became familiar with BHR/PVC and the PVO community and its needs. In addition, BHR/PVC and CSTS worked together to determine how they could effectively manage the results-oriented contracting. Under new CSTS leadership, the contractor demonstrated its capacity to manage for results and meet PVC and PVO needs. It has developed and put into practice administrative procedures and management systems to implement CSTS effectively. On its part, USAID has managed this performance-based contract skillfully. PVC/CS, together with the Office of Procurement, have focused on the five results. When it appeared that the project was off track and was not achieving project objectives, USAID brought this to the attention of the contractor, and together they made the changes required to produce the desired results. PVC has expected CSTS to act independently and creatively to achieve what was expected, and the contractor has EXECUTIVE SUMMARY iii reacted positively. CSTS has been well managed fiscally as well—its burn rate is very close to where it should be at this point in the project. Findings Based on the findings of the evaluation, CSTS performance has been good and BHR/PVC should exercise the option for years four and five of the contract. CSTS staff has understood PVO and BHR/PVC needs for technical assistance and responded to them. Project staff was instrumental in developing the joint CSTS/CORE web site that has been widely appreciated and used. CSTS staff has provided extensive support to the CORE working groups, especially in the development of workshops. The staff, in conjunction with BHR/PVC, coordinated a significant update of the technical reference materials (TRMs) and improved the KPC by developing, with CORE, the KPC2000+. CSTS has designed and tested a new tool to determine the organizational capacity of health units that has been found useful by the PVOs that have used it. The contractor has supported BHR/PVC by responding to a variety of needs, including logistic support of the application and Detailed Implementation Plan (DIP) reviews. The evaluation team first reviewed the five results and the associated 25 minimum acceptable performance standards (MAPS) to determine appropriateness and relevance. The results of the analysis identified 13 performance standards that were appropriate and should be retained unchanged, 2 that required rewording, 9 that should be modified due to changes in the project or to make the targets realistic, and 1 that is redundant and should be dropped. (see complete MAPS and recommendations in annex I) In addition to the findings related to the results/MAPS, the following six crosscutting issues and themes were explored: Utilization of Technical Assistance The child survival grantees are a heterogeneous group. The analysis of the CSTS request log found that the small PVOs located outside the Washington, DC area utilize CSTS less. This group requires a different kind of support than the larger PVOs based close to Washington. After an initial needs assessment, CSTS primarily based its activities on an analysis of the applications, DIPs, and midterm and final evaluations to determine areas of weakness and the need for technical assistance. To improve the participation of the PVOs that are currently only minimally engaged, CSTS should ß become more proactive and reach out to the small and distant PVOs; ß facilitate the formation of a caucus for small PVOs; ß undertake a brief annual review of each PVO, with PVC and CORE, to identify those in greatest need of technical assistance; and, MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT iv ß set priorities for support and capacity-building efforts to the most needy PVOs. Field Orientation Most of the interview respondents pointed out that the vast majority of CSTS support has been provided to the PVO headquarters level. While this builds capacity at the headquarters that should be transferred to the field offices and partners, it does not happen enough. To improve the situation, CSTS should ß concentrate its energies on capacity building at the regional and country levels and focus on field office and partner staffs while holding more workshops in the field. Training/Capacity Building The one-time workshops had virtually no follow up to evaluate, reinforce, and refresh the training that had taken place. The evaluation team identified a need to develop a means to build core competency in both interventions and management. This would help CSTS respond to the differing needs of the PVO community, as well as the problem of frequent staff turnover. To improve the capacity building element of CSTS, the contractor should ß carry out a study of PVO staff turnover, identifying causes and ways to reduce turnover or deal with the repercussions; ß add training expertise with knowledge of such topics as curriculum and materials development to help develop innovative approaches; ß conduct a feasibility study relating to the establishment of distance learning courses in the core competency areas; and, ß if feasible, develop a pilot module based on the KPC 2000+ to test the viability of distance learning courses. Materials and Dissemination The short, operationally oriented periodicals and technical publications, such as Bookmarks!, are most popular and seem to fill a niche. The CSTS and CORE web sites are also utilized frequently and are highly appreciated. The Institutional Strength Assessment tool to monitor organizational management capacity has been favorably received by the PVOs that have used it to date. Because more than 40 percent of the current grants are in either French- or Spanish-speaking countries, and to date CSTS has translated only a few items, there is an expressed need for more translation of what CSTS produces. To improve performance, CSTS should EXECUTIVE SUMMARY v ß determine the most important publications/materials that will be continued, and ß translate documents, materials, and tools into French and Spanish. Linkages with Cooperating Agencies Although collaboration with the CAs is not a part of CSTS’s scope of work, the projects managed by CAs are often involved in innovative technologies, making them valuable partners in the effort to decrease infant/child mortality. The PVOs (especially the more mature ones) are interested in introducing the new interventions in their projects, while the CAs operationlize their cutting-edge interventions and are encouraged to collaborate with PVOs. Thus, both the PVOs and CAs would benefit from closer collaboration. In order to facilitate the relationship between the PVOs/CORE, PVC, and the CAs, it is recommended that CSTS ß in close collaboration with CORE, proactively liaise with the CAs involved in child survival and encourage them to appoint a specific person to serve as the contact person with the appropriate working group. Strategic Planning There is a need for PVC/CS, CORE, and CSTS to define clearly what has to be done, identify available resources, and determine roles and responsibilities for achieving CSGP objectives. This is particularly important at this point when the leadership of PVC/CS is about to change. It is recommended that ß CSTS, PVC, and CORE carry out a joint strategic planning exercise with the new chief of PVC/CS; and, ß PVC address the question of generic management support (e.g., database and web site development and maintenance, organizational development) to the PVC Office in its forthcoming strategic planning effort. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT vi 1 I. INTRODUCTION A. PURPOSE The purpose of this midterm evaluation of the Child Survival Technical Support (CSTS) contract (FAO-C-98-00079-00) is to determine whether the contractor’s performance warrants the United States Agency for International Development’s (USAID) exercising the option to continue for years four and five. CSTS is a performance-based contract. As stated in the evaluation team’s scope of work (annex A), the evaluation has the following two objectives: ß To assess the performance of the CSTS project in order to provide USAID’s Bureau for Humanitarian Response, Office of Private and Voluntary Cooperation (BHR/PVC) with information on whether and/or how to continue funding for the last two option years of the contract, and make suggestions for improving the work performance under the contract. ß To assess the current performance-based contract to determine whether the contract performance standards should be modified to better achieve the Strategic Objective of BHR/PVC. B. EVALUATION TEAM This evaluation was conducted under the Monitoring, Evaluation and Design Support (MEDS) project. Two consultants were hired to work with a representative of USAID’s Bureau for Global Programs, Field Support and Research, Center for Population, Health and Nutrition (G/PHN). The team leader has over 30 years of experience in international health and nutrition programming and evaluation. He has been associated in a number of different ways (e.g., proposal and Detailed Implementation Plan (DIP) reviews, project design, and evaluation) with the Child Survival Grants Program (CSGP) since its inception 16 years ago. The child survival specialist on the team also has considerable experience with the child survival program, having worked for a grantee for several years and having assisted other PVOs with developing proposals and DIPs for child survival grants. In addition, the child health advisor for G/PHN’s Child Survival Division, who is also one of the co-managers of the Basic Support for Institutionalizing Child Survival (BASICS) II project, participated with the team as her time permitted. C. METHODOLOGY This evaluation was carried out from February 27 through April 6, 2001, in accordance with the statement of work (SOW) (annex A) developed by the Child Survival Division of PVC (PVC/CS). MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 2 The CSTS evaluation used a modified participatory approach that involved representatives from each of the stakeholder groups, including BHR/PVC, CSTS, Johns Hopkins University (JHU), Child Survival Collaboration and Resources Group (CORE), cooperating agencies (CAs), USAID’s G/PHN, PVOs, MEDS, and the evaluation team members. Each provided valuable input into determining the evaluation methods, selecting the respondents, and identifying the crosscutting issues and themes. This input was collected through a series of team planning meetings, a startup stakeholders’ meeting, and close telephone/electronic mail communication between the evaluation team and PVC/CS. A set of meetings was held at the conclusion of the data collection process with PVC/CS, CSTS, and the entire stakeholders’ group to debrief them on the evaluation’s findings and recommendations. Field site visits were not included in the SOW. Initial plans were to develop an indepth questionnaire that would be distributed via electronic mail to each of the PVOs who have been involved in the PVC/CS–funded program. Past experience, however, has shown that the response rate to electronic mail questionnaires to the PVOs has been low, ranging from 20 to 40 percent. Attempts at trying to balance competing needs for collecting qualitative input with that of making the questionnaires as easy to use as possible, proved to be unrealistic.1 Instead, a different strategy was undertaken, which has arguably produced a richer data set than could have been achieved through use of a questionnaire. Finally, the following three strategies were used for the collection of data: ß interviews with key informants, ß distribution of a miniquestionnaire, and ß review of project-related documents. Interviews with Key Informants Interviews were used to collect qualitative data on the CSTS project to date and on possible future directions. The crosscutting issues identified by the evaluation team and discussed with the stakeholders formed the basis for the interviews. However, respondents were given broad latitude to go beyond the questions and to explore and contribute new areas or topics for the team to pursue. Both telephone and in-person interviews were conducted. When feasible, other respondents were invited to participate in the interviews so that they would be similar to minifocus group discussions. Interviews with 56 key informants were held with persons from ß BHR/PVC (9), 1 The timing of this evaluation presented a significant challenge to data collection. Late March is one of the busiest times of the year for the PVOs with DIPs due March 31. This is confounded by PVOs being inundated with questionnaires and surveys that they are always being asked to complete. This led to an offer by BHR/PVC to extend the March 31 deadline by two weeks, so that the PVOs would have time to participate in the evaluation. This level of flexibility was greatly appreciated by many of the PVO participants in this evaluation and noted as a strong motivating factor for their participation. I. INTRODUCTION 3 ß G/PHN (3), ß CORE board members and working groups (5), ß CSTS/MACRO (9), ß CAs (4), ß JHU (1), and ß PVOs (23). Two consultants to the project were also interviewed. To ensure a broad representation and to compare different types of PVOs, categorization and selection was based on ß size (2 small, 2 large), ß length of time in the CSGP (2 new, 2 old), ß location of their headquarters in relation to Washington, DC (2 near, 2 far), and ß frequency of contact with CSTS (2 frequent, 2 infrequent). In addition, two PVOs that are members of CORE were included that are not currently receiving child survival funding from BHR/PVC. All of the above PVO interviews were conducted by home office staff. Fortunately, it was also possible to interview one PVO field staff member who had participated in CSTS–sponsored trainings and who had seen several of the documents that they have produced. (See annex B for a complete list of the people interviewed.) The evaluation team developed a set of questions on the crosscutting issues that were used to structure the interviews (annex C). A questionnaire was then designed based on this set of questions for use in the PVO interviews to ensure that consistent information was collected (annex D). Analysis of the interview results was done by debriefing within the evaluation team after the interviews and then reviewing the meeting notes to identify common themes and new areas that had not been previously considered. These were then incorporated into the questionnaire in the form of follow-up probing questions. Distribution of a Miniquestionnaire A miniquestionnaire was developed to give all stakeholders who were not involved in the interviews an opportunity to provide input into the evaluation (see annex E). The questionnaire included four open-ended questions; it was sent via electronic mail to the PVC/CS’s list serve. Responses came from four PVOs, including three that incorporated input from their field programs. Review of Project Related Documents The evaluation team also reviewed a substantial number of project-related documents, including reports, proposals, materials/tools developed by CSTS, and past evaluations (see annex G). Particular attention was paid to the CSTS contract, the performance work MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 4 statement, and quarterly reports and their use as a monitoring tool for project results and activities. The project’s five results and 25 minimum acceptable performance standards (MAPS) were reviewed and assessed based on their relevance to the current needs of the clients (CORE, PVOs, and PVC/CS) and how they are being monitored (see annex I). Annex J provides the schedule of the evaluation. 5 II. BACKGROUND A. CHILD SURVIVAL GRANT PROGRAM The Child Survival Grant Program (CSGP) began in 1985 as part of USAID’s effort to address the high rates of infant and child mortality and morbidity. The U.S. Congress earmarks funds each year to reduce the more than 10 million child deaths that occur annually in the developing world from immunizable diseases, diarrhea, acute respiratory illnesses, malaria, malnutrition, and human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS). A portion of each year’s budget is allocated to U.S.– based PVOs as a means of effectively implementing child survival strategies in the field through their networks of country offices and local partners. Over the years, the grants to PVOs have been found to be a cost-effective way to implement innovative community￾based programs to reach the most vulnerable segments of the population, improving their health and nutritional status. In the early years of the program, the PVOs were asked to focus their attention and efforts on the two primary interventions of child survival: oral rehydration therapy (ORT) and immunization. The overall purpose of the CSGP was to develop the technical capacity in the recipient PVOs in child survival in general and in these two interventions in particular: how to design a good project, how to implement it, how to improve the knowledge of the beneficiaries and change behaviors, and how to evaluate whether progress was being made. In most cases, the implementing PVO was involved directly in the delivery of services to beneficiaries. The PVOs were focused mostly on the project itself and learning how to promote effectiveness. In the early 1990s, JHU’s Child Survival Support Project (CSSP) developed the knowledge, practice, and coverage (KPC) survey instrument to enable the PVOs to determine the nature and extent of the child health problems in their project areas and to monitor project progress. The grants were limited to three years with a budgetary guideline of approximately $500,000, with a 25 percent match requirement provided by the PVO. The CSGP has evolved over the years. In the new millennium, the child survival grants are awarded for as long as five years and can be funded for up to $1.3 million. Additionally, instead of being limited to two interventions, there are now eleven choices: ß nutrition, ß breastfeeding, ß immunization, ß control of diarrheal disease, ß pneumonia case management, ß control of malaria, ß maternal and newborn care, ß child spacing, MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 6 ß prevention of sexually transmitted infections/HIV/AIDS, ß other infectious diseases, and ß integrated management of childhood illness (IMCI). Child survival grantees are no longer restricted to the KPC survey, but have a number of alternative tools to measure and track progress. (These tools will be discussed in the course of this evaluation report.) In recent years, the matches from the PVOs receiving grants are often more than the 25 percent minimum. Also, the focus of PVO child survival efforts has broadened beyond the projects themselves to include district, national, and even international perspectives and initiatives. While change was occurring in the PVO world, changes were also taking place in USAID. There was little technical expertise in PVC/CS in its early years. The office was primarily concerned with the proposal and DIP review process, which was supported and managed by an outside contractor. As the agency underwent reengineering and emphasis was placed on achieving and reporting results, the composition of the PVC/CS changed. Beginning in the mid–1990s, technical child survival specialists were recruited to provide support to the child survival grantees. Johns Hopkins fellows were assigned to the office to work with the grantees to ensure that their performance was improving and support was provided to enable them to achieve their objectives and results. Simultaneously, in 1996, PVC developed its first Strategic Plan with one Strategic Objective (SO): to build capacity in the grantees and their local partners so that they not only achieve their technical intervention objectives, but also improve their organizational effectiveness in terms of program design, sustainability and financing, management and logistics, monitoring and evaluation, and quality assurance. This signaled the shift from a focus largely on the more technical/intervention specific issues to the more process-oriented focus of building management and organizational capacity issues. During the first 15 years of the CSGP, 337 projects were funded by USAID at a cost of almost $210 million. Added to this is another $110 million in matching funds provided by the PVOs. The total annual amount allocated by PVC/CS has varied between a low of $7.3 million in 1986 to almost $18.5 million in 1997. Thirty-five PVOs have received grants and over 184 million children under the age of 5 and nearly 25 million women of childbearing age have benefited from the program. At present, PVC/CS is supporting 79 projects in 36 countries that are being implemented by 27 different U.S.–based PVOs (see annex K). B. TECHNICAL SUPPORT CONTRACTS In the first few years of the CSGP, a need was identified for ongoing technical support to PVC/CS in the administration of its grants and to the PVOs in their implementation. The result was the Child Survival Support Project (CSSP), which was awarded to Johns Hopkins University. For 12 years, JHU/CSSP provided the technical underpinnings for the USAID/PVO child survival effort. It provided BHR/PVC with technical assistance in developing and administering the process by which PVO child survival program applications were requested and assessed and implementation plans were reviewed. For II. BACKGROUND 7 the PVOs, CSSP provided a range of technical support, including training in the use of the KPC data collection methodology, specialized workshops and conferences, technical reports, and individual technical assistance to PVOs in project design and implementation. As the nature of the CSGP changed in the mid–1990s, PVC/CS believed that there was a need to modify the technical support contract as well. There was less of a need for child survival intervention expertise, as it had grown in the PVOs since the beginning of the program. Instead, there was a need for support of the grantees to strengthen their organizational capabilities. This led to an important addition to the child survival request for application in 1997, inviting PVO grantees to carry out the Discussion-Oriented Self￾Assessment (DOSA) to test it and be part of a cohort to measure capacity. Many of the original PVOs dropped out since the DOSA could not measure change. With the agency’s emphasis on results and management/capacity building, PVC saw the need for a modification in the type of technical support it and the PVOs required. PVC envisioned the next support contract as being considerably more participatory, working closely with the PVOs and PVC/CS. As part of the reengineering process and the increased focus on results, USAID introduced a new contracting mechanism. It was referred to as performance-based contracting and it held the contractor accountable for achieving specified results while allowing it broad discretion in how those results would be achieved. This new form of contracting was being introduced in USAID as CSTS was being conceptualized. While many offices were reluctant to come up with indicators and specific standards that would permit this new type of contracting, PVC/CS was interested and determined to adopt and apply it to the CSGP. It was the first unit in BHR to design a performance-based contract and one of the first in the entire agency. The CSTS contract was awarded to MACRO International in September 1998. The contract identified five results, corresponding indicators, and 25 MAPS. JHU was a subcontractor on CSTS from the beginning. There were other changes in the environment at the same time. One important development was the role of the Collaboration and Resources for Child Survival (CORE) group made up of all the PVOs that had ever participated in the CSGP. This organization was in its start up phase when MACRO assumed responsibility for the technical support to the CSGP. CORE was identified in the request for proposal (RFP) and CSTS contract, but it was impossible at that time to foresee how fast CORE would develop, how instrumental it would be in inter–PVO collaboration, and how CORE and CSTS might interrelate. In addition, the role and significance of the CORE working groups, formed in April 1999, was not envisaged when the CSTS RFP was issued in January 1998. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 8 9 III. TECHNICAL FINDINGS AND RECOMMENDATIONS A. SUMMARY OF FINDINGS After a thorough review of the CSTS contract since its inception in September 1998, it is recommended that the fourth and fifth year option be exercised. The performance of the contractor has been satisfactory, especially since the beginning of 2000. The PVC/CS, its colleagues at CORE, and the child survival PVO community are pleased with the support they have received from CSTS. In this section, the findings of the CSTS midterm evaluation are described and ideas and recommendations for future directions and activities are raised. B. CONTRACT RESULTS: FINDINGS AND RECOMMENDATIONS Overview As mentioned in the previous section, the overall approach of the performance-based contract is to provide the contractor with broad discretion in identifying and selecting the most appropriate strategies for reaching the contract results. For each result, a set of four to six indicators and minimum acceptable performance standards were identified by PVC/CS. The performance standards provide CSTS with a measurable target that, if achieved, should lead to the result. These were developed inhouse at PVC/CS and went through five revisions. As this was their first experience using this type of contract, PVC/CS had no past models for guidance. The contract performance work plan consists of 5 results, 21 indicators, and 25 minimum accepted performance standards (MAPS). These are provided in annex I, along with recommendations specific to the performance standards. Progress on meeting the contract results and their respective performance standards is tracked through CSTS’s quarterly written reports and meetings held between CSTS and PVC/CS staff. It is important to note that CSTS’s quarterly reports track progress on the performance standards and not directly on the indicators. This is because the performance standards provide specific targets whereas the indicators do not. For instance, the first indicator under result 2 is, “PVO knowledge of the information dissemination system.” The related performance standard is, “within nine (9) months of the date of this contract, all child survival application program (CSAP) PVO project officers shall have knowledge of the information dissemination system.” A summarized version of how the evaluation team first conceptualized the contract results and performance standards is shown in figure 1. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 10 Written Materials Produced and Information Dissemination System Established Figure 1: Contract Results, Indicators, and Performance Standards Results Indicators/Performance Standards (summarized) 1. CS Technical and Management Expertise 2. Technical CS Information 3. Project Management Tools and Skills 4. Organizational Development 5. CSGP Management by PVC Technical Assistance and Training Provided to PVOs (1i-1iii) CORE/CSTS Consultant Database Enhanced PVO Monitoring and Evaluation Capacity Strengthened Management Training of PVOs and NGOs Assessment and Training of PVOs in Organizational Development PVOs Providing Technical Assistance to Local Partners PVOs Increase Knowledge in CS Sustainability Strategies HMIS System for Review of PVO Submissions Program Summary Analysis and Ad Hoc Requests III. TECHNICAL FINDINGS AND RECOMMENDATIONS 11 Discussions with both PVC/CS and CSTS highlighted the fact that there was initial confusion regarding CSTS’s understanding of the overall intent of the results-oriented contracting process. CSTS’s proposal presented an appropriate program plan to address the results; however, work plans were evidently more difficult to develop. This was because the distinctions between the five results are not clear cut, and a close reading of the performance standards shows that several support multiple results. For instance, the performance standards for technical assistance listed under result 1 are necessary to support each of the five results. It became apparent to CSTS and the evaluation team that the reality was more complex than that presented above. The evaluation team developed the following diagram to demonstrate the reality of the integrated approach (figure 2). This complexity is compounded by the constantly changing environment in which the project operates. There have been several major changes over the past three years since this contract was enacted: ß the development and expansion of information technology has created new ways to transfer knowledge and develop skills; ß the effectiveness of CORE and its working groups as key players in the global health community has added a major player to the mix with whom CSTS and PVC/CS have been able to work in order to more effectively reach their mutual goals; ß the transition in focus of the CSGP away from direct services to capacity building of PVOs and their local partners has led to a fundamental change in the types of training and capacity building the PVOs require; and ß the continual raising of standards as the science and practice of child survival continues to develop has created a wider range of training and capacity building needs within the PVO community. The remainder of this section provides findings and recommendations specific to each of the five contract results. A final summary section, and the accompanying annex I, provides additional findings and recommendations on the contract’s performance standards. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 12 Figure 2: Contract Results Per Reality Results Indicators/Performance Standards (summarized) HMIS Program Summary Analysis and Ad Hoc Requests 1. CS Technical and Management Expertise 2. Technical CS Information 3. Project Management Tools and Skills 4. Organizational Development 5. CSGP Management by PVC Technical Assistance and Training Provided to PVOs CORE/CSTS Consultant Database Enhanced Written Materials Produced and Information Dissemination System Established PVO Monitoring and Evaluation Capacity Strengthened Management Training of PVOs and NGOs Assessment and Training of PVOs in Organizational Development PVOs Provide Technical Assistance to Local Partners PVOs Increase Knowledge in CS Sustainability Strategies System for Review of PVO Submissions III. TECHNICAL FINDINGS AND RECOMMENDATIONS 13 Result 1: Child Survival Technical and Management Expertise The contractor shall be responsible for measurable improvement in the knowledge and skills of PVO headquarters and field staff of child survival PVO programs currently funded under the CSAP and, to a limited extent, other USAID sources. Findings CSTS has provided technical assistance to the PVOs on an individual basis and through their support of the CORE working groups. As of March 7, 2001, 570 requests for technical assistance had been received and responded to by CSTS staff. The most frequent requests have been for: ß monitoring and evaluation (140), ß written materials (69), ß assistance on the new KPC 2000+ (48), ß organizational development (43), and ß questions specific to the various child survival interventions (101). These requests are recorded in the CSTS request log, where they are tracked based on the PVO, type of request, length of time required to respond, and date. They are reviewed monthly by CSTS staff and through the customer satisfaction surveys. PVO satisfaction was found to be high with this service, which is consistent with the results that CSTS has noted in its survey. CSTS has played a key role in the support of the CORE working groups, which also falls under the heading of technical assistance. In the telephone interviews with PVOs and working group participants, there was consistent appreciation expressed for their work— particularly with the monitoring and evaluation, malaria, and IMCI groups. CSTS staff involvement has helped to ensure necessary continuity, momentum, and technical expertise to these groups, which are volunteer driven by PVO staff with very limited schedules. The refining of the CORE/CSTS consultant database has been done in close collaboration with CORE and is a good example of how the two groups have worked together to achieve a shared objective. PVO knowledge about and use of this database is high, based on the customer satisfaction survey conducted by CSTS and the midterm evaluation interviews. Currently, CSTS is not equipped to track the number of hits made to this database or to quantify how many consultants have been hired by CORE PVOs using this list, other than through the annual customer satisfaction survey. The option to set up a system that would track hits is feasible and could provide a picture of the level of usage. As of December 2000, 114 consultants were listed in the database. Currently, there is no system for deleting consultants from the list when they are no longer available, which would make finding available consultants easier for the PVOs. CSTS also has explored options where both consultants and PVOs could post comments about their experiences working with each other directly into the database, in an effort to MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 14 increase the quality and satisfaction with consultants. However, most of the PVO respondents reported that they would check references before hiring a consultant regardless of what information was provided in the database. The database does allow space for consultants to list contact names and electronic mail addresses from their past assignments that can be contacted for references. Further, many organizations are growing increasingly concerned about the potential liability resulting from evaluations and recommendations of past employees and contractors, especially when they are being put in writing in a public forum, such as a database. The potential exists for using other means to ensure the quality of consultants, such as a certification system where consultants that complete training in a particular topic are officially certified to provide that service, whether it is the Institutional Strength Assessment (ISA) tool or designing an HIV/AIDS prevention program. This is discussed further in section III.C. on training/capacity building. To help in their searches for appropriate consultants, some of the PVOs suggested that it would be useful to reassess the different fields that can be used for database searches, adding in the newest technologies and skills (i.e., ISA methodology or the new KPC 2000+) as they are developed. Recommendations ß Institute an annual re-registration requirement, where each consultant receives an electronic mail message automatically from CSTS requiring them to respond if they are interested in remaining in the database. ß Annually review the list of “areas of experience” in the database that can be used for searches, incorporating new technologies and skills for which PVOs will be seeking consultant services. (Additional result 1 recommendations that are specific to performance standards are included in annex I.) Result 2: Technical Child Survival Information The Contractor shall be responsible for creating and operating an information dissemination system with the following minimum features: ß Accessible by PVO headquarter and CSAP project staff to the latest developments related to the state of the art in child survival programming; ß Accessible by PVO headquarter and CSAP project staff to published and unpublished materials on lessons learned and best practices in child survival programming from grantees and others; and ß Capable of handling the rapid exchange of PVO–identified information and PVO–generated requests for information on child survival issues. Findings CSTS has focused on the development of the CSTS interactive web site and the preparation of written materials and tools for dissemination to the clients to reach result III. TECHNICAL FINDINGS AND RECOMMENDATIONS 15 2. The CSTS web site is well known and very popular throughout the PVO community at both the headquarters and, where accessible, the field level.2 Those who are not accessing the web site report that it was due to lack of time. The fact that many of the document files are available in portable document format (PDF) is appreciated because of the ease of downloading them in the field. This has also greatly expanded their availability, not only for the child survival PVOs, but to other PVOs, nongovernmental organizations (NGOs), local government agencies, students, and academics. The ability to access the PowerPoint files from past workshop presentations is also popular. In addition to using them to review the presentations for those PVOs who cannot attend the workshops, some have been downloading them so they can replicate the presentations when they visit the field. The one part of the web site that could be strengthened is the calendar, which tends to be incomplete and is not kept up to date. Out of town PVOs, especially, said that this is important information for scheduling their travel. CSTS has consistently been providing new approaches to addressing long-standing challenges. It is now developing a database interface that can be used for the new KPC 2000+. While CSTS is well ahead of schedule in its production of written materials, the State of the Art (SOTA) papers have required more time than originally envisioned. This has been due to the participatory approach taken, which has involved a broad group in the selection of topics (CORE working groups, faculty and students at JHU, the CAs) as well as writing and reviewing the papers. CSTS’s subcontract with JHU requires three SOTAs per year, totaling nine for the first three years of the contract. (It should be noted that this requirement only exists between JHU and CSTS, not between CSTS and PVC/CS.) Currently, three have been finished (A Manual of Manuals on Nutrition Interventions for PVOs, Methodology and Sampling Issues for KPC Surveys, and Reaching Communities for Child Health and Nutrition: PVO Contributions to Community IMCI). Topics and writers for the remaining six have been identified and are at various stages of development. The subcontract between CSTS and JHU was not finalized until April 1999. Since the transition in leadership at CSTS, the three principal parties (CSTS, JHU, and PVC/CS) have stayed in close communication. PVO satisfaction with the SOTAs has been high for those that are directly involved with the specific SOTA topic (implementing a nutrition program or beginning a KPC survey), less so for those who are not. Discussion on how these topics are selected is included in section III below. Among the periodic publications, Bookmarks! is very popular. It is a brief, newsy update on latest directions, developments, tools, and materials that is distributed via electronic mail to the child survival community on an irregular basis, as needed. As of December 2000, 65 have been produced and distributed. People like it because it is short, easy to read, and informative. Back issues are available on the CSTS/CORE web site. Feedback from the target audience indicates that CS Connections is less valued. While seen as interesting and well produced, it is not long enough to give program people sufficient operational details (i.e., ingredients of success or how-to guides) that would enable them 2 While CSTS is able to track the number of hits on its site, it cannot distinguish which pages on the site are being accessed. It is also not possible to tell the location of those using the site, since the tracking program only identifies the location of the server. Since AOL (a popular internet service provider) is located in Northern Virginia, a disproportionate number of users are there as well. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 16 to replicate or implement the featured approach and it takes precious time to read. Therefore, the benefit of the publication after its first two editions has yet to be identified. Recommendations All recommendations relating to result 2 are incorporated into annex I and in section III.C, Crosscutting Issues and Themes. Result 3: Project Management Tools and Skills The Contractor shall be responsible for measurable improvement in the Child Survival Application Program (CSAP) staff skills in collecting, analyzing, and presenting child survival project information. Activities may include provision of technical assistance, training, networking, or mentoring services. Findings The principal activities for achieving this result include training workshops; review of PVO applications, DIPs, and program evaluations; editing PVC/CS guidelines; providing technical assistance to PVOs that are preparing their DIPs; and the development of monitoring and evaluation tools, including the Participatory Program Evaluation Methodology (PPEM) and the new KPC 2000+. The PVO community has expressed broad interest in the new KPC 2000+ and its accompanying components, including the Rapid Catch, Lot Quality Assurance Sampling (LQAS), and the Field Guide (currently in development). CSTS staff have participated in field trials and convened a 3–day meeting on the new tool in October 2000. Both were appreciated by the PVOs involved. The one concern raised through the PVO interviews related to the type and amount of training required for PVOs to be able to implement the KPC 2000+ on their own. Some PVOs expressed interest in returning to the older JHU– developed 2–week training of survey trainers approach. However, PVC/CS and CSTS both have noted that that course did not in fact train participants in how to train others in its implementation. Further, this model is not viable financially, considering the number and distribution of PVO staff requiring training, estimated at 20–30 projects per year implementing KPC surveys (10–15 baseline and 10–15 final). CSTS’s approach to establishing PVO capacity in the implementation of the new KPC 2000+ is through the KPC Field Guide. It is hoped that this guide will be descriptive enough so that the reader can conduct a KPC survey without having attended a KPC training workshop. Current plans are to have it presented at the April 2001 CORE Group annual meeting for review by the PVOs. (This discussion is continued in section III.C, Training/Capacity Building.) The PVOs that have participated in the design, testing, and use of the PPEM, as well as the workshops, have found it to be a useful approach and tool for involving partners and other key stakeholders in program design, monitoring, and evaluation. Some of the PVOs that have used the PPEM believe that it should be shortened and made more cost effective. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 17 The technical resources materials (TRMs), appropriately subtitled by one PVO as “The Cliff Notes of Child Survival,” have received excellent reviews from the PVO community. The TRMs were greatly strengthened by adding more components to the management/organization sections and extensively updating the individual child survival interventions. This is a USAID document, yet CSTS staff participated in its updating along with the CORE working groups, the CAs, and USAID health specialists. One issue specific to the TRMs is the size and number of figures. This makes the file difficult to download from the web site, especially in the field. Recommendations All recommendations relating to result 3 are incorporated into annex I and in section III.C, Crosscutting Issues and Themes. Result 4: Organizational Development The Contractor shall be responsible for measurable improvement in the PVOs health units’ ability to (a) plan and direct their health portfolios, including the development of the skills of health unit headquarters and field staff in such areas as negotiation, conflict resolution, human resource development, team building, leadership, strategic planning, financial and administrative management, sustainability and management of donor and constituency relations; and (b) develop the above mentioned skills with NGOs in developing countries. Findings The ISA tool, focusing on PVO headquarters health unit capacity, has been developed and field tested and is available on the CSTS web site. The contract required that an institutional assessment methodology be developed and submitted to the cognizant technical officer (CTO) within the first 6 months of the project. A participatory approach to the design of this methodology was developed within six months of CSTS’s hiring of an organizational development specialist in February 1999. This approach required more time to implement because it broadened the network of people and organizations involved. The ISA was developed by CSTS after reviewing existing organizational capacity assessment tools (e.g., Organizational Capacity Assessment [OCA], DOSA, and Global Excellence in Management [GEM]). It has been piloted with one PVO and three others are initiating their assessments. Those PVOs that have been involved in the design, through their membership on the monitoring and evaluation working group and in the field tests, have voiced strong support for the tool and the way it was developed. Some PVOs have adapted it to use in their field programs with their local staff and partners. (This discussion is continued in result 5, where the ISA database is covered.) The issue of sustainability continues to raise intense interest and debate within the CS community. Related issues raised in the PVO interviews included identifying the true financial costs of sustaining project activities at all levels (family, community, health facility, private sector, and health care system) and the development and linking of capacity-building and sustainability objectives. The CORE–CSTS Sustainability Initiative is charged with developing a common framework that will assist PVOs in MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 18 program design, implementation, and phase over to local ownership. The CORE-CSTS Sustainability Initiative began with the Sustainability Dialog in Calverton, Maryland, at MACRO (March 2000) and has since involved research (including other sectors of development, such as ecology/land management), interviews, and PVO surveys on critical issues and project self-assessments. Additional progress and field testing are required before the value of, and contribution made by CSTS’s sustainability work can be determined. Recommendations All recommendations relating to result 4 are incorporated into annex I and in section III.C, Crosscutting Issues and Themes. Result 5: CSAP Management by PVC The contractor will assist BHR/PVC in improving their monitoring capability of the CSAP. Findings Since its focus is primarily on services provided directly to PVC/CS, result 5 is the most discrete and linear of the five contract results. It focuses on the development of the health management information system (HMIS) for PVC/CS, assisting in the application and DIP review process, responding to ad hoc requests from PVC/CS, and providing an annual summary analysis of the CSAP grant program. An inherent challenge in maintaining the three HMIS databases is dealing with the inconsistency of the data provided by the PVOs. While the data for the project database comes from the DIPs and is thus a mechanical function of transfering content from documents into the database, this information can change at any time. If the PVO does not take the initiative to report these changes to CSTS, the database will become dated. The PVOs reported that they are using the project database to identify where organizations are working, especially when they are deciding which countries to focus on, and later when they begin doing their initial site assessments. In addition, some have found it to be a useful tool for learning about other PVO approaches to CS. Conversely, the ISA and child survival indicator databases are both voluntary, with the PVOs encouraged, but not required, to submit data. Data collection for the ISA database will likely be consistent when CSTS is directly involved in its implementation. However, as more and more PVOs are able to implement these assessments without CSTS’s participation, ensuring comprehensive data collection will become more difficult. A similar challenge faces the child survival indicator database. Discussion on the need to collect a standardized data set to monitor the global impact of the child survival program dates back to the development of the KPC methodology by JHU. The rapid catch indicators are the latest approach adopted to address this need. CSTS has worked closely with the CORE monitoring and evaluation working group to select the indicators that each project will be asked to collect and report on at baseline and final. This collaboration III. TECHNICAL FINDINGS AND RECOMMENDATIONS 19 between CSTS and the monitoring and evaluation working group has been key to ensuring PVO participation. The monitoring and evaluation working group is now actively promoting the use of the rapid catch indicators throughout the PVO community, which is a far more effective incentive than having PVC/CS require them. However, the problem is that initial results will not be available until 2004, when the final evaluations are completed. The possibility of using currently available child survival data should be explored as an interim strategy. This will likely require the assistance of a biostatistician, which CSTS does not currently have on staff. Overall satisfaction with CSTS’s response to the ad hoc requests and the system it has developed for the application and DIP reviews has been very positive. PVC/CS is averaging about three to six ad hoc requests per quarter that range from requests for additional copies of reports to historical data justifying cost per beneficiary calculations—which can require substantial time to complete. Together with PVC/CS staff, CSTS has systematized and, to the extent possible, streamlined the document review process. One ongoing concern is the amount of staff and consultant time required for the reviews. Yet the value of the exercise is significant because it supports the other results as an important measure of change in PVO capacity. PVC/CS satisfaction with CSTS’s first annual child survival grants review (1998–1999) has also been largely positive. It demonstrates CSTS’s solid understanding of the CSGP, the PVOs, and their projects, especially for a first effort. While valuable recommendations are being provided about the PVOs, suggestions were made by PVC/CS that it would be useful to also include a section that would give broader, more visionary recommendations for the CSGP overall. The length of time required to complete the annual review has taken longer than first envisioned due to a number of factors. The amount of time required to review all the documents (midterm and final evaluations, DIPs, and applications) is significant, especially as the CSGP has become increasingly complex. These documents usually do not all come in on time, which further delays the process. CSTS has been good about raising these issues with PVC/CS as they have surfaced and PVC/CS has agreed to a later deadline. It is also important to note that CSTS has provided preliminary analysis of performance in time for the Results Review and Resource Request (R4) preparation. Recommendations ß CSTS needs to develop an interim strategy for utilizing the currently available child survival indicator data set to determine the impact of the child survival programs that can be used until the rapid catch data are collected. ß See performance standard 5iii in annex I for the other result 5 recommendation. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 20 Summary Findings on the Contract Results and Performance Standards The evaluation team has reviewed the contract and the project documents, including annual work plans, quarterly reports, workshop evaluations/summary reports, and the CSTS technical assistance request log. The purpose of conducting this analysis was threefold: ß to assess whether CSTS has been meeting the terms of the contract, ß to help frame the strategic planning process being recommended as a result of this evaluation, and ß to provide lessons learned for those who will be developing similar contracts in the future. Based on this review, the overall finding was that the contract results were being met by CSTS. However, in doing its review, the evaluation team also found that some of the 25 contract performance standards need to be redefined (see annex I). The results of the analysis identified the following: ß 13 performance standards that were appropriate and should be retained unchanged (1i, 1iii, 2i, 2ii, 2iv, 3i, 3vi, 4ii, 4iv, 4v, 5i, 5ii, 5iv); ß 2 that required rewording (1iv, 3iv); ß 9 that should be modified due to changes in the project or to make the targets realistic (1ii, 1v, 2iii, 3ii, 3iii, 3v, 4i, 4iii, 5iii); and ß 1 that can be dropped due to redundancy (2v). Summary Recommendations ß A review of the results, indicators, performance standards, and recommendations provided in annex I should be carried out by CSTS and BHR/PVC/CS as part of the strategic planning session recommended in section III.C below. ß The terminology used to label various groups needs to be consistent throughout the contract and reports to ensure clarity. For instance, does “PVO headquarters and field staff” (result 1) equate to “CSAP PVO staff” (performance standard 1ii) or to “PVO HQ and CSGP project staff” (result 2)? ß The wording of the results in the quarterly reports needs to be made consistent with the contract (i.e., contract results 1, 3, and 4 state that “CSTS shall be responsible for...,” while the same results in the quarterly reports state that “CSTS shall demonstrate...,” or “CSTS shall show...”). III. TECHNICAL FINDINGS AND RECOMMENDATIONS 21 C. CROSSCUTTING ISSUES AND THEMES After reviewing the results and performance standards as defined in the CSTS contract, and after interviewing numerous individuals familiar with CSTS, the evaluation team identified a number of crosscutting issues and themes that deserve special attention and discussion. Some of these issues and themes have been the subject of ongoing consideration since the CSGP began in the mid–1980s. This is not the first time attention is being focused on several of these topics, but that does not lessen their importance or the need for giving them serious attention, especially since the environment has changed. The following crosscutting issues and themes are discussed: ß utilization of technical assistance, ß field orientation, ß training/capacity building in core capabilities, ß materials and dissemination, ß linkages with child survival–related projects managed by CAs, and ß strategic planning. Utilization of Technical Assistance In the first year of the CSTS contract, a needs assessment was carried out and was published in August 1999. It surveyed the child survival projects and their PVO headquarters. The purpose of this 5–month exercise was to identify common PVO needs so that the CSTS project could develop its plans for providing technical assistance to the PVOs. The needs were identified according to project attributes such as region, funding year, and relative experience of the PVO’s headquarters in the CSGP. The needs expressed in the 1999 needs assessment can be grouped into the following categories: ß monitoring and evaluation, ß child survival interventions, ß health and management information systems, ß organizational development, ß information dissemination, ß networking, and ß DIP development. The child survival grantees are a very heterogeneous group with each subgroup requiring very different types and levels of support. This evaluation found that the distant and small PVOs utilized CSTS assistance less frequently. This section examines how CSTS has attempted to determine PVO needs and steps that it could take to assume a more proactive role to assist those most in need and that are currently underutilizing CSTS assistance. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 22 As a result of this survey, CSTS confirmed that the PVOs in the CSGP are a heterogeneous lot. For example, they found that the newly funded PVOs required more technical assistance and capacity-building support, while the larger, more established PVOs wanted help in documenting their operations and successes, launching innovative interventions, and carrying out operations research that would advance the field and their own technical capacities. There are several ways to categorize the PVO community: size (large versus small), location (distance from Washington-near versus far), experience (old versus new) and current enrollment in the CSGP. For years, USAID has asked PVC about graduating the larger, more experienced PVOs from the CSGP. The term graduation is no longer in vogue, but the idea remains. The concept of mentoring was introduced in the mid–1990s, but to date has only been used once. When asked why, the large PVOs reported that it was not worth their while—requiring considerable time with minimal compensation. Another suggestion that was raised during the evaluation was to separate the mature PVOs and give them block grants to develop local partners in the countries in which they work. Another way that CSTS assesses PVO need is through the program review process. This occurs on an annual basis. It has resulted in a very helpful and thorough document, extracting the lessons learned from the 1998–99 CSGP year. PVO applications, DIPs, and midterm and final evaluations are reviewed and trends are identified. Recently, it was found that projects are often narrowly focused. For example, only a few of the grantees that have immunization components include vitamin A distribution in their operations.3 Such feedback makes the CSGP a learning project—always building on its experiences and identified shortcomings. The ability of CSTS to provide this perspective is one of its most valuable contributions. It complements CORE, which is primarily implementation oriented. This approach also provides a form of needs assessment by allowing CSTS to identify performance trends and gaps and make them the subject for workshops, publications, or changes in PVC/CS guidelines. A less obvious but equally influential barometer of PVO needs and interests is the membership of the working groups, which are volunteer driven and therefore based on the perceived needs of each participant. Individuals vote with their feet to join the IMCI, reproductive health and safe motherhood, nutrition, behavior change and communications, malaria, monitoring and evaluation, or the newly forming management working group. This not only shows where PVO interests lie, but also drives the decision-making process that determines the topics CSTS, CORE, and PVC/CS will focus on next. CSTS’s 1999 PVO needs assessment suggested that in the future an assessment of needs be incorporated into all its activities so that a formal survey will not be necessary each year. This, in fact, is what the project has done. But it is difficult to ascertain how successful this approach has been. Those interviewed were not always clear how 3 However, many PVOs include vitamin A as part of their immunization programs, especially as part of National Immunization Days. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 23 decisions about products or workshops were made—where the demand came from or how it was ascertained. An example from the first year of the project was the participatory evaluation workshops that were carried out regionally in Haiti, Bolivia, and Senegal. While there was a general PVO interest in the subject, it was suggested that the decision to hold the workshops was at least partially supply driven—the consultant and tool were available and CSTS took advantage of it. The question that is most often raised relates to how the different needs of the PVOs involved in the CSGP can be met. For example, the newer grantees typically need more technical and management support as they launch their projects. The older, more mature groups are eager to introduce state-of-the-art strategies and approaches to their projects. They also are interested in establishing their credibility and linking with the multilateral donor agencies. The latter group of PVOs tends to have the more senior and available spokespersons who are able to represent them at meetings and participate more fully on the CORE board and working groups. Thus, their ideas would appear to have a better chance of being heard and acted upon over the often quieter voices of the smaller, newer, and distant PVOs. The evaluation team also analyzed the types of PVOs that were making the most frequent (more than 10) requests for technical assistance from CSTS. The PVOs were grouped by ß length of time in the CSGP (old being those funded prior to 1997), ß location (near is in or around Washington, DC), and ß size (large having more than one child survival project support staff at headquarters). This was then compared with the number of requests made by each group of PVOs for technical assistance. The results showed that the newer, more distant, and smaller PVOs were seeking less assistance than the more established, closer and larger PVOs. This simple analysis of the CSTS log book shows that about half of the old, and a majority of the new, grantees are infrequent users of CSTS technical assistance (figure 3). The disparity increases when the location or proximity of the PVO to Washington is considered. Only a few of the PVOs located outside the Washington area avail themselves of CSTS’s services (figure 4). This compares with almost 75 percent of those whose offices are located in or close to Washington. The greatest disparity is in the size category, where only a few small PVOs contacted CSTS for support while a majority (71 percent) of the large agencies utilized CSTS (figure 5). MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 24 Proximity to Washington, DC 0 5 10 15 20 Near to DC Far from DC Number of PVOs Frequent Infrequent n=27 Note: Old is defined as all PVOs who were in the child survival program before October 1, 1997 Figure 3: Frequency of Use: New Versus Old Figure 4: Frequency of Use: Far Versus Near Note: Near is defined as the Washington–Baltimore–Richmond area Length of Time in CSTS Program 0 5 10 15 20 Old New Number of PVOs Frequent Infrequent n=27 III. TECHNICAL FINDINGS AND RECOMMENDATIONS 25 Figure 5: Frequency of Use: Small Versus Large Size 0 2 4 6 8 10 12 14 16 Large Small Number of PVOs Frequent Infrequent n=27 Note: Large is defined as more than one headquarter/regional support staff To help address the challenge of reaching the newer and smaller PVOs so that they voice their needs and have them addressed, it is recommended that a caucus of the small PVOs be established to share common experiences, discuss needs, and advocate for those needs to the PVO community at large, CORE, and PVC/CS. It is interesting to note that at the last annual CORE meeting in Phoenix, there was a meeting of the small PVOs. However, this was more for the purposes of providing a forum for the new people to meet each other; there was no follow up (e.g., a list serve being developed or communications), nor has the group been formalized. The interview respondents raised the need for CSTS to become more proactive and take the initiative in the provision of technical assistance. While CSTS initiates contact with the newly funded PVOs that are starting their DIP process, there are no requirements for the PVOs to connect on a one-to-one basis with PVC/CS and/or CSTS from the time that their DIP is approved until the end of their grant. One way to ensure that each of the PVOs is staying connected to the CS network is to institute a brief annual review of existing child survival grants by PVO. The project officer and technical advisor in PVC/CS assigned to each PVO could meet with the person at CSTS who is knowledgeable about that specific PVO’s activities. It would be done once each year, providing an opportunity to review the PVO’s performance and identify specific technical assistance needs. For its own management purposes, CSTS has found it beneficial to divide the PVO community into groups of four to five PVOs and assign each member of its team to one of these groups. Because it has been primarily for internal tracking purposes, not all the PVOs may know who their CSTS contact person is. Any special need should be referred to the appropriate specialist (e.g., a question about evaluation referred to the monitoring and evaluation advisor), however, the CSTS member responsible for that particular PVO would be more familiar with their country operations, the history of their child survival grants, and problems that were raised in their DIPs. For the annual review, the participation in CSTS activities would be noted. Did someone from the PVO attend a MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 26 workshop? How many times did they request technical assistance and for what purpose? Have they followed up on the recommendations pointed out in their DIP reviews? The purpose of the annual review would be to initiate a process whereby PVC/CS and CSTS could identify those PVOs that have needs but are not utilizing its services. The three to five PVOs experiencing problems, or those that have been out of contact, would be targeted for special attention. CSTS would reach out to the contact person at the PVO and offer support and assistance to strengthen the PVO’s performance. This focused, proactive approach should improve overall PVO performance in child survival programming by directing support to the groups with the greatest needs. In addition, it would help cut across the various segments of PVOs, reaching out to those that may not approach CSTS despite a need to do so, such as the small and distant ones. Because the results of the CSTS contract are very broad and general, it has been difficult for the contractor to develop an overall strategy. In the beginning, no mechanism was in place for determining the needs of the PVO community or segments of it. Therefore, to some, the focus of the materials and workshops appeared at times to be arbitrary and ad hoc. Activities did not always seem to be driven by a set of predetermined priorities. The need for and the advantage of a strategic planning exercise is discussed below and will help in the needs assessment process. (See recommendation 13). Recommendations 1. CSTS should prioritize its technical assistance and training strategies in favor of PVOs with the greatest needs. To do this, CSTS, with PVC/CS, should institute a brief annual review of each PVO to identify those PVOs with the greatest need. In addition, CSTS should segment the PVO audience and monitor PVO technical assistance usage in order to be more proactive at reaching out to the newer, smaller, and more distant ones. 2. CSTS, with the CORE group, should consider facilitating the formation of a caucus of small PVOs to determine and advocate for their special needs. Field Orientation Another concern frequently voiced by those involved in or familiar with the CSGP is the concentration of support resources on headquarters staff and its operations. The technical and management capacities that are built are most often at the PVO home office. At the admission of CSTS, the PVO field offices and their local partners receive very little direct benefit from the CSTS project. One reason for this is the lack of resources which makes travel a rare opportunity. In the first two and a half years, the only regional While the vast majority of CSTS support has been directed at PVO headquarters level, the evaluation found that most CSGP participants want emphasis to be placed on building capacities at the field office and local partner levels. How CSTS can accomplish this is the challenge to any contractor providing support to the CSGP. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 27 training sessions were the three participatory evaluation workshops and the quality maternal and newborn care meeting, which was held in Nairobi. Several more are planned for later in the third year (capacity building in Haiti, safe motherhood in Central America, and LQAS in Asia). This is the direction that CSTS and many of the PVOs want to take, especially the larger, more developed groups. Heterogeneity issues again present themselves, with the smaller PVOs continuing to value the conferences held for support staff in the United States, both for their content and for the opportunities that they afford for networking. It is hoped that a caucus of small PVOs—described in recommendation 2 above—will help to meet their need for networking and that the capacity-building recommendations will address how headquarters staff might be accommodated. CSTS is also interested in increased exposure to the field so that it can become more familiar with the implementation issues that the project must address. However, with the limited resources available to the project, it is not clear that additional travel would be the best use of scarce resources. If future workshops are held, CSGP sites in the area should be visited as occurred when CSTS workshops were held in both Bolivia and Kenya. Evidence from the interviews and program reviews suggests that support and capacity building is especially needed and important at the field office and local partner levels. The question is how to do that in a cost-effective manner. New information technologies, as they become available in the United States and in the field, will be a major factor in answering this question. (This is discussed further in the next two sections on training/capacity building and materials dissemination.) For the short term, the current strategy of regional workshops seems to make the most sense. One area that is not being addressed by CSTS and that is not on its agenda is the often referred to challenges associated with scaling up the child survival project activities. In many cases, the PVOs do an outstanding job and produce impressive results among the project population. All too often, the effort stops there. While it is unrealistic to expect the PVOs themselves to greatly expand their coverage, it is possible for them to work with the government in their respective countries to incorporate their newly developed child survival approaches and strategies to strengthen existing public programs. Attention to linking successful local projects with public efforts and expanding them to reach a larger portion of a needy population is receiving great attention by the international health community (e.g., the World Health Organization) and featured in USAID projects (e.g., BASICS II). Foundations are also interested in exploring the role PVOs can play in facilitating the scaling up process. CORE and CSTS might consider collaborating on this in the future. Recommendations 3. CSTS and the follow-on CSGP support contractor should focus its capacity building at the regional and country levels and on field office and partner staffs. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 28 CSTS capacity-building activities to date have included materials development/dissemination, provision of technical assistance, and holding workshops. The latter have involved limited follow up. The needs of the various categories of PVOs span a wide range of training needs. However, there is a general need for a core competency (involving both technical interventions and management capability) in all child survival grantees, especially throughout the smaller/newer PVOs and the field program levels of the larger/more established PVOs. The problem of institutionalizing capacities should be addressed by exploring distance learning technologies. The CSTS team should include someone with expertise in curriculum development and training methodologies/techniques, who is skilled and conversant in these new technologies. 4. Additional workshops should be held in the field while greater efforts are made to develop new ways to build capacities by using innovative technologies (see recommendation 9) to address the needs of all grantees, large and small. Training/Capacity Building in Core Capabilities CSTS training activities have been largely focused on a dozen workshops that have been developed and conducted primarily assisting CORE working groups. It is clear when speaking with CORE members that the workshops either would not have taken place or would have been much less effective had it not been for CSTS involvement. The most valuable contributions made by CSTS were in agenda development, facilitation of the workshops, and payment of some of the costs. The quality and value of the workshops is difficult to ascertain. Some participants found them very helpful, while others did not. The only feedback that CSTS sought was the post-workshop evaluation sheets filled out by participants and through document review (i.e., using DIP reviews to assess capacity of the PVOs to collect and utilize KPC data after the KPC workshop), the results of which are difficult to link back to the training. For the most part, participant satisfaction with the training workshops has been very favorable, usually being rated between very good and excellent. However, there is no evidence of follow-up surveys to determine the effectiveness of the training. Without this longer-term feedback, it is very difficult to improve the training. The lack of follow up is also a concern with respect to refreshed or maintained capacity. Field research, such as IMCI training in Zambia (Pond 1998), shows that the effect of training does not last very long and that knowledge is lost quickly and practices revert to previous habits if nothing is done to reinforce what was learned in the workshop. Other training programs, such as Wellstart, followed up their training by providing the participants with quarterly or semi annual packets of articles or guidelines that refreshed or built on what had been taught during the training. As a capacity-building project, one would expect that CSTS would have significant staff expertise in training. Currently, while there is an organization development specialist on the team as well as persons who can facilitate training, there is no specialist with expertise in curriculum development and training methodologies/techniques. CSTS III. TECHNICAL FINDINGS AND RECOMMENDATIONS 29 trainings to date have been directed at building individual knowledge and skills, which is an important aspect of strengthening institutional capacity. It is hard to know what can or will be achieved through these single workshops alone. The networking benefit is something that is appreciated by all the attendees, but the impact on capacity building is difficult to identify. Because of the high staff turnover rates in the PVOs, there are limitations in training individuals who may or may not be there a year after the workshop is presented. The critical question is whether the capacity of the PVO institution is increasing. From speaking to those involved in the proposal and DIP reviews and by reading the program review, the answer is: “not necessarily.” The quality varies considerably from year to year and even the quality of proposals or DIPs from the same PVO in the same year. Those interviewed will relate how they have to hire consultants to carry out the KPC because they no longer have the trained staff to implement it. In some ways, some of the PVOs have regressed. Thus, there is a need to seriously address the question of how to build, institutionalize, and sustain capacity in the individual PVOs. One point made by some PVOs was that performance measures never show improvement because the CSGP continually raises the standard—constantly adding interventions and demanding more. Although PVC/CS only expects the PVO to focus resources on the most important problem(s) in the proposed CS project area, the PVOs seem to perceive that they need to do more than that. Each year the request for proposal includes something new. While this is true, there are still a number of basic technical and management competencies with which, if the feedback on the quality of the applications and DIPS is to be believed, some of the PVOs continue to struggle. As noted above, rapid staff turnover makes organizational capacity building difficult. There are any number of reasons for this phenomenon, but no one knows much about the cause. One PVO said that the impact of losing a key support person or field program manager can be traumatic, especially for the small PVOs. When that person leaves, they often take with them half the organizational memory. On the other hand, PVO headquarters and field staff that have left usually continue to work in international health, often with other PVOs, ministries of health, or donors, using the CS skills they acquired in the CSGP. Others become consultants and support the grantees. Nonetheless, because the issue was raised often during the evaluation, it is suggested that a study be conducted of a group of individuals who have dropped out to learn more about the phenomenon. Why did they leave? What are they doing? What could be done to reduce the problem? It might be helpful to study those PVOs that have experienced little turnover to see if some attributes and practices can be identified and passed on to others. The CSTS project staff has discussed among itself how some of the new technologies could be used to solve this long-standing and fundamental capacity-building challenge. However, the exciting electronic technologies now available have not been applied or utilized to improve the institutionalization of capacity in the PVOs. These technologies were not even imagined when the CSTS contract was in the design phase. The Internet has been utilized well by CSTS with its multifaceted and much used web site. However, MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 30 it has not yet been used for training per se. Nor has there been any employment or serious discussion of the CD–ROM technology in CSTS. Respondents suggested several ideas that are worth considering. One possibility is the development of a series of short core courses covering both the technical interventions and the management topics that would be available electronically, accessible through the Internet and/or CD–ROM. This core curriculum would include instruction/lectures, be linked to guidelines and other materials, and have exercises and exams. It could be operated as a distance-learning program. There would be a support person (who could be at a school of public health or even a CORE working group4 ) who would be available to answer questions and grade the exam.5 In the long term, this function could find a permanent home, linked to an organization or academic institution. Once a person has passed the exam they would receive a certificate indicating that they are qualified in that particular field. This would allow PVC/CS and the PVOs to relate to a certain level of expertise, knowing the exact skills and expertise of an employee, student, job candidate, or consultant. It would establish core competency among all those involved in implementing child survival projects. One of the concerns raised during interviews with PVOs was the amount of time that field health staff have to be away from their regular project responsibilities to attend conferences, training workshops, and intraorganizational meetings. Although these are important and useful activities, they often represent a real cost to a project. Such a system of long distance electronic-training would allow students to go through the courses on their own schedule and would be useful when there are changes in positions, new staff are hired, or a new intervention is added. This would help address the need for institutionalization of these skills and expertise that has plagued many of the PVOs and the CSGP since its inception. While some of these new technologies are appealing and seem intuitively to be the best approach, there can be considerable costs associated with their design, production, and dissemination. If the decision is made that this is the best way to proceed, either additional resources have to be invested in the support contract, or some of the current activities (possibly including some publications and workshops) will have to be reduced or eliminated. These are not easy or pleasant choices to have to make. Concern was raised about consultants and how they can be kept current. Consultants are rarely included in any of the CSTS workshops, except occasionally as resource people. However, if the PVOs are going to continue to rely on consultants as they do now, there is a need to provide them with access to this training. By making the electronic courses available to them, there is some confidence that they can be kept up to date on the latest developments in a particular aspect of child survival programming. It is even possible to 4 Because the CORE working group members are already overextended, their involvement would only be possible if there were some way to compensate them for their time and efforts so that the responsibility would not be added to what they already are doing. 5 Objective exams can be graded automatically, as is already being done by Project Hope. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 31 envision the consultants paying the respective fee to access the training course in the specialty in which they want to become proficient. CSTS should carry out a feasibility study of a distance learning approach, identifying alternatives that have been used in the health and other sectors, identifying mechanisms, determining costs, and identifying feasible technologies at the regional and field levels. Specialized technical knowledge and experience will be required to conduct such a study. A pilot effort could be designed and tested if time and resources are available. With up to 30 PVO projects implementing KPC surveys every year, one priority that has been voiced by the PVOs is training of field staff in the new KPC 2000+. The KPC 2000+ field guide is now reaching a final draft stage and could be adapted into a CD– ROM version, with the addition of exercises and examinations. Conceivably, it could be tested prior to the end of September 2003 when the CSTS contract comes to an end. Recommendations 5. CSTS should collaborate with CORE to identify studies that have been done and/or implement a study on the high staff turnover rates experienced by the CS PVOs. 6. Training expertise, specifically in curriculum and materials development, should be added to the CSTS team for years four and five. 7. CSTS should carry out a feasibility study (including cost) of designing and maintaining distance learning courses on the technical interventions and management issues included in CSGP. 8. A distance learning module on the KPC 2000+ should be developed and pilot tested. Materials and Dissemination The list of publications and tools produced by CSTS can be found in annex F. These have been disseminated in printed format, as well as made available on the CSTS/CORE web site. Publication costs have been kept to a minimum by limiting the number of copies printed. For example, the project prints only 500 copies of CS Connections. Moreover, distribution costs are minimized by sending the copies for the field programs The challenge of identifying the most useful topics and the media easiest to use to transfer that information from CSTS to the PVOs and to the field programs will continue to be an ongoing challenge for the project. The shorter, operationally oriented periodicals, and the larger, technical publications and tools, are highly valued by CSTS clients. PVOs have found the web site developed and maintained by CSTS to be very helpful. Efforts should be expanded to increase PVO access to translated materials, especially in French and Spanish. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 32 to the PVO headquarters to be forwarded to the field and by synchronizing the completion of documents with the semiannual meetings where PVOs gather, such as CORE’s annual meeting in April and the RFA orientation meeting in October. This is a good way to get packets of materials to the target audience while at the same time creating awareness about and interest in the products. CSTS has developed a widely appreciated and frequently used web site. It has combined efforts with CORE in such a way as to benefit both organizations. It is becoming increasingly apparent that the Internet will be the favored method of communication in the years to come. Problems still exist when trying to get information to the field since connections are often unavailable, of poor quality, and very slow or expensive to access. Moreover, downloading large attachments can be difficult and, in many cases, impossible. Despite this, CSTS is encouraged to create hyperlinks6 to as many of the reports and operationally helpful tools and guidelines as possible, such as the Health Facility Assessment (HFA), to maximize access by the PVOs. Additionally, CSTS is encouraged to limit the use of unnecessary graphics in the electronic versions of their written materials to reduce the time it takes to transmit and download documents. CSTS has monitored and assessed the utility and quality of the tools that it has helped to develop (the KPC 2000+, PPEM, and ISA) through its participation in field trials and by following up directly with the PVOs involved. One of the challenges the project faces is to find efficient, low-cost ways to assess and monitor the quality and utility of its publications, such as Bookmarks!, CS Connections, and the SOTA papers, at both the headquarters and field levels. As stated in the results section, PVO satisfaction with the quality of these products has been high; however, satisfaction with their utility is less consistent. The current mix of these publications needs to be further explored by CSTS to ensure that each is necessary and includes media that is easy to use. Documentation of the PVO’s efforts and methods is also seen as very important, but the PVOs have neither the time nor the money to do it. They are too action oriented to be able to devote their scarce personnel and financial resources to such an activity. It requires a lengthy, operationally oriented publication, such as the four studies on noteworthy PVO activities produced by BASICS I. However, the latter are very costly and beyond the financial resources available to CSTS. Despite the need and CSTS’s effort to document lessons learned in CS Connections, the CSTS midterm evaluation found limited support for the relatively new publication because it is not detailed enough to enable project directors to replicate what is described. Moreover, there are other similar publications. The majority of PVO headquarters staff members interviewed by the evaluation team reported that they were being inundated by information in the form of newsletters, journals, list serve electronic mails, and new tools, etc. It is becoming increasingly difficult to select useful material and discard the rest, resulting in a large number of documents that often are not read. On the other hand, the PVOs were not interested in having an outside source determine what they should or should not receive. Rather, they 6 There are numerous hyperlinks to technical instruments and guidelines in the TRMs. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 33 prefer to continue to make these decisions themselves for their own offices and for the field. The challenge for most PVOs comes when they need to find the appropriate tool or information source to meet their own unique situation and needs. This is a role that CSTS has played since the beginning of the contract and with which the PVOs who have used their services have been pleased. However, there are still PVOs reporting that they need someone to help them make informed decisions on appropriate tools particularly because of the higher standards being set by new organizational capacity assessments, health facility assessments, and the mass of other tools available. PVO interest in having additional access to translated materials and tools was a common theme through many of the interviews. With almost half of the existing CSGP projects located in predominately French– (15), Spanish– (14), or Portuguese– (4) speaking countries, this is important if capacities are to be built at the national, district, or community levels. Other than the participatory evaluation publication, which was released in English, French, and Spanish, CSTS has not translated any of its other documents. However, at least one of the PVOs found Bookmarks! so valuable that they translated it for its field offices, while another PVO has invested in translating the TRMs into Spanish. CSTS should take advantage of this and arrange with the PVOs that have translations to upload them to the CSTS/CORE web site for easy access by others, or provide a database on the web site where PVOs can announce when they have translated materials they are willing to share. CSTS maintains a repository of documents relating to child survival. This permits it to refer PVOs or others to documents that they might not be able to find elsewhere. There is nothing in CSTS’s scope of work about serving as a clearinghouse and it has no desire to become one. However, it does see one of its functions as being a broker or facilitator— connecting those who need a resource with appropriate contacts. This is a useful and highly valued role for CSTS. Recommendations 9. CSTS should review publications along with the PVO community to determine the optimal use of resources. 10. CSTS should review its web-based publications to make sure that they are linked to related materials whenever possible and to review all the materials that are available through the Internet to ensure that the files are as small and easy to transmit/download as possible. 11. CSTS should ensure that French and Spanish versions of the most important publications and reports (e.g., Bookmarks! and technical guidelines and tools) are available on CSTS/CORE web site. Or, a database should be established on the CSTS web site where PVOs can find out which child survival materials have been translated by other PVOs and how they can access them. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 34 Linkages with CS–related Projects Managed by Cooperating Agencies (CAs) To date, most of CSTS’s involvement with the CAs that manage USAID G/PHN projects has been in the review of the DIPs and contributions to the TRMs. However, there has been limited involvement by some of the health-related projects in the work of PVC/CS and CSTS/CORE. In fiscal year 2001, PVC/CS allocated resources to the following five most appropriate and interested projects to foster a close working relationship: ß Change, ß Environmental Health Project (EHP), ß Food and Nutrition Technical Assistance (FANta), ß Quality Assurance Project (QAP), and ß Rational Pharmaceutical Management (RPM). This initiative was based on the reasonable premise that the contracts and agreements for these child survival projects included the provision of working with PVOs. This was a means to facilitate that relationship. After almost 6 months, only three of the five have nearly completed memoranda of understanding (MOU) with PVC, outlining how they will support the CSGP. The CAs have difficulty appreciating how to work effectively with the PVOs and how to utilize the $50,000 allocation from PVC (limit for this year) without investing a lot of their time and effort. The problems faced in getting the MOUs accepted is indicative of the difficulty of getting the PVOs and the CA–managed child survival projects to work together. At times, there are tensions between the two groups. The CAs are viewed by some as large organizations that use large amounts of the Congressional child survival set-aside funds and do not really understand how PVOs operate. In addition, the PVOs have seen many presentations by the CAs describing their projects. Moreover, the time limitations and the difficulty of coordinating with the CAs make collaboration difficult. From discussions with individuals from the PVOs and the CAs, it is obvious that there are significant and substantial mutual benefits to be derived from collaboration. In fact, both groups need each other. On the one hand, the child survival projects managed by CAs need the PVOs to implement their activities and be effective at the community level, a particular strength of the private voluntary sector. BASICS II recognized the role of PVOs in the launching of community IMCI and provided funding to CORE, worked The child survival projects managed by CAs and the Global divisions that oversee them are a resource for CORE since they are involved in innovative activities and provide a means to infuse innovation into PVO projects. Simultaneously, the PVOs provide the CAs with a means to put into operation what they found effective in their field research. Efforts are needed to encourage the CA projects to work more collaboratively with CORE and CSGP. While recognizing that CSTS does not have a contractual obligation to relate to the CAs, CSTS could take advantage of their unique position to facilitate collaboration. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 35 closely with CSTS and the IMCI working group, and organized a well-received workshop on the subject in early 2001. On the other hand, the PVOs need access to the expertise and cutting-edge technologies and interventions being tested and documented by the CAs’ child survival projects. The latter can be seen as field laboratories, developing and testing the efficacy of new techniques and methodologies. The PVOs can upgrade their projects and operations by introducing some of the CAs’ innovative approaches developed in the child survival projects, while the CAs’ child survival projects would benefit by taking their interventions to the next operational level, serving larger populations and encountering some of the common constraints faced with any field operation. This would move the science from a controlled environment to an operational one before it becomes large scale and affects most of a region’s or country’s population. There would also be a benefit if the Global divisions involved in child survival programming and responsible for overseeing the CA–managed child survival projects were to establish working relationships with the relevant CORE working group. For example, if advances are made in immunization, the PVOs should know about them and integrate the new information into their immunization efforts in the child survival projects. In order to realize the synergies and mutual benefits that appear to exist if CAs’ child survival projects and PVOs collaborate, there is a need to bridge the gap that currently separates them. CSTS might be able to help. Although CSTS has no contractual obligation to work with the child survival projects and the CAs that manage them, it could facilitate bringing the two groups together. CSTS seems well qualified to perform this function since it has worked with both the PVOs and the CAs, understands their capabilities as well as their needs, and has earned their confidence. It is difficult to work with the CAs’ projects since they operate independently; there is no collective of CAs that serves the function that CORE does for the PVOs. This, of course, is not an easy matter since the projects managed by the CAs are in place for only the length of their contracts. However, there is some continuity and the benefits would appear to outweigh the constraints. If each CA were to appoint one person as their child survival liaison, these representatives could attend the meetings and keep their respective CA apprised about the CSGP while informing the small group of CAs that work in child survival and others of their own activities that could be integrated into the child survival program.7 These individuals would serve as the initial connection between the child survival projects and the CORE working group. Recommendation 7 EHP has appointed a full-time liaison officer to link with all outside groups (e.g., Pan American Health Organization, WHO, United Nations Children’s Fund, World Bank, and CORE). MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 36 12. In collaboration with representatives from the CORE working groups, CSTS should facilitate a proactive liaison with the most appropriate child survival projects managed by CAs. Strategic Planning The implementation of the CSGP has been greatly improved over the last five years under the leadership in PVC/CS, CORE, and more recently in CSTS. They have worked exceptionally well together to build the capacities of the child survival grantees. While the overall objectives driving and directing the CSGP have been clearly defined, there has not been a clear definition of roles and responsibilities. Given the current individuals in charge of the three components of the program who have a shared understanding and vision, this lack of definition has not caused a problem. However, the distinction between the roles of CSTS and CORE are unclear. The two organizations have worked together to achieve a common goal without devoting much time to attribution. This has been helped by the fact that the project director of CSTS has not insisted on having a high profile for himself, his project, or his firm. While this has worked well to date, there is no guarantee that it will continue to be the case. This is of particular concern now that the current chief of PVC/CS is due to retire in September and the project officer, who is responsible for day-to-day operations, departed in March 2001. This turnover could be very disruptive unless an effort is made to clearly identify and come to consensus on roles and responsibilities, and tasks and activities to be accomplished. In other words, a joint strategic planning exercise is required. The possibility of CORE assuming the responsibility for the technical assistance functions currently handled by CSTS was raised during the course of this evaluation. This should be considered in the strategic planning effort. One of the reasons against such a move is the inability (because of an obvious conflict of interest) of CORE to be involved in the application and DIP reviews. In addition, activities such as database and web site development and maintenance and organizational development are not part of the comparative or competitive advantages of the PVOs or CORE. While it has been noted that they could hire the talent, the advantages of having a group that already has these capabilities as part of its core competencies will have to be considered. A strategic planning exercise including PVC/CS, CORE, and CSTS is a good opportunity to identify priorities and come to consensus on how scarce resources will be allocated in There is a need for PVC/CS, CORE, and CSTS to clearly define what has to be done, establish priorities, identify resources, and determine roles and responsibilities for achieving the objectives of the CSGP. If this exercise could be done during the turnover process to the new chief of the PVC/CS, it could help increase the possibility that the successful operation that has developed over the last two and a half years would continue to grow. Moreover, PVC needs to decide how it will access the needed generic management support it requires. III. TECHNICAL FINDINGS AND RECOMMENDATIONS 37 the last two years of the CSTS contract. This visioning process will also give everyone the chance to fit CORE more officially into the child survival strategy. When CSTS was launched less than three years ago, CORE was a fledgling organization. To give some idea of its growth, its budget has grown from $150,000 in 1998 to $850,000 for the current year.8 Moreover, it will soon be an independent entity once it completes its registration as a 501(c) 3 nonprofit organization in the next few months. In light of the turnover that will soon take place in PVC/CS, the timing of this recommended strategic planning exercise is important. It would be beneficial to all three groups if the current chief of PVC/CS participated in the planning. In addition, it is advisable to arrange the strategic planning to include the person that will be taking her place. It is understood that the new chief of PVC/CS has been named. Even if she is not able to take on her position full time until after the current chief departs, it would be extremely important for her to participate in the planning sessions on a temporary basis. This will help her to become fully aware of all aspects of the program and also help to foster her sense of ownership, since she will be responsible for its implementation over the next several years. A good facilitator should be engaged for the strategic planning exercise and at least 3 days should be set aside so that there is no shortage of time to think through all aspects of the CSGP and explore all possibilities. At the same time, PVC is about to begin its own strategic planning exercise that will determine its direction for the next five years. As a part of this exercise, PVC should consider the role of the CSTS contract in relation to other support contracts for CS and the rest of the office. Some of CSTS’s activities at present are appropriate and needed by other programs in the office, such as Matching Grant, Farmer-to-Farmer, and Food for Peace. To date, the programs have been vertical and self-contained. There may be an advantage in having one technical group handle all the technical management support, including database and web site development and maintenance and organizational development. The CSGP will still need its special technical support which might look very different from the way it looks now—including such responsibilities as facilitating CA and PVO collaboration, developing and supporting a distance training program, and providing technical intervention expertise. Recommendations 13. PVC/CS, CORE, and CSTS should carry out a joint strategic planning exercise, if possible with the participation of the new chief of PVC/CS, to determine roles and responsibilities, and available resources for achieving program objectives. 8 Four hundred thousand dollars of the total is from PVC, $300,000 from G/PHN/CS, and $150,000 from the Africa Bureau. The last two contributions are for community IMCI and are programmed to the relevant working group. CORE also has $8 million for polio operations that are currently being programmed through World Vision until CORE is a legal independent organization. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 38 14. During its forthcoming strategic planning effort, PVC should address how some of the generic support needs of the office can be provided most cost￾effectively. 39 IV. FINDINGS ABOUT PROJECT ADMINISTRATION A. CSTS STARTUP There was a 6–month gap between the end of the CSSP II contract and the beginning of the CSTS contract. When the MACRO–managed contract began in October 1998, it started slowly. The performance-based contract mechanism was new to all and no one was quite sure how it should work. The cognizant technical officer (CTO) for the CSTS project had a vision that the contractor would be guided by the performance work statement in MACRO’s contract. USAID did not expect or want to micromanage the contract or to tell the contractor how to achieve the results. However, the project director for CSTS was more focused on responding to PVC/CS’s leadership than in identifying how CSTS was going to achieve project objectives and support the PVOs implementing child survival grants. There were some communication problems, which became critical with the submission of the second year work plan. The work plan was not linked to the five results that were the basis of the CSTS contract; in fact, the CSTS project director unilaterally changed the results. The original staffing of the CSTS project was also a concern. Two staff had transferred from CSSP. While this may be seen as a way to provide some continuity in the transition from one contract and contractor to another, it proved the opposite. The two individuals in question had not been in leadership roles in the CSSP. PVC/CS was intent on the new project having a highly participatory, empowering process. In addition, although each of the original key staff had worked directly for a PVO, individuals who knew and interacted with CSTS at that time reported that they “were not familiar with or sensitive to the PVO culture.” There was very little collaboration during the first year of CSTS with CORE and PVOs on the development of work plans and activities. The CSTS staff on the project from the beginning also noted that leadership within the project was lacking. This, combined with USAID’s concerns about the lack of leadership and vision in the CSTS contract, came to the attention of the MACRO hierarchy. The second annual work plan made it clear that a change in project director was required if the contract results were to be met. In early 2000, the contractor appointed a new project director. The first project director was a technical specialist. It soon became clear to PVC/CS that someone with strong management skills was required. While some would say that a contractor cannot win a procurement without a technical person in charge, experience in this case would suggest that a person who understands and is skilled in management can often be more effective in a capacity-building, support contract such as CSTS. The new project director was originally the organizational development specialist on CSTS. Early in 2000, he was made interim project director. However, within a month, all agreed that he was the preferred person to manage the project. It did not take long for the trust and MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 40 confidence among the three partners (PVC/CS, CORE, and CSTS) to be established, and has persisted ever since. The CSTS has developed and put into place effective financial management procedures, systems, and support facilities. The staffing pattern is appropriate for the work currently being undertaken by the project. MACRO has considerable experience in managing USAID contracts, having been the contractor of such large procurements as the Demographic and Health Survey (DHS) contract for more than a decade. PVC/CS believes that it has benefited from the fact that MACRO has a range of skills that can be called upon to broaden and deepen CSTS support to the office and the PVOs, such as database development, web site development and management, and organizational development. In general, staff morale is good despite heavy workloads and the constant pressure of responding to predetermined and spontaneous needs for technical support. Staff members respect project leadership and respond well to it. However, the staff cannot be expected to assume additional responsibilities. If any of the new activities recommended in this report, or emanating elsewhere, are to be added, something will have to be eliminated. The strategic planning exercise discussed and recommended in Section III is meant to inform this process. B. USAID MANAGEMENT With the new leadership installed at CSTS, relations improved between the contractor and USAID. They were then able to communicate effectively and maintain a clear understanding of each other and the client’s wants and needs. The new project director changed a few personnel and assigned existing staff as most appropriate. He hired a new technical specialist with PVO and extensive field experience. The most interesting and important aspect to document is the way that USAID/PVC and the Office of Procurement (OP) has managed the CSTS contract. As mentioned previously, there was no past experience to draw upon when it came to designing and implementing a results-based contract. The evaluation team was told that the performance standards went through five drafts. However, instead of paying too much attention to these standards, PVC/CS focused on the results. They wanted the contractor to be free to achieve the results in the most creative and cost-effective means possible. By asking the contractor to meet certain standards that may or may not be appropriate, the contractor would have to spend considerable time and resources in a less than optimal manner. By giving the contractor flexibility on achieving the specified results, USAID has enabled the CSTS project to operate effectively and creatively. The expected results that CSTS would achieve were clearly specified. What was not specified was how they were to achieve the results. For example, it was not specified that CSTS should work with CORE, but the contractor appreciated that it could never realize result 1 without a strong working relationship with CORE. CSTS did not have the resources or personnel to IV. FINDINGS ABOUT PROJECT ADMINISTRATION 41 provide individual PVOs with technical assistance. However, by working with the technical working groups in CORE, CSTS was able to assist the child survival PVO community. Of course, the CORE working groups also needed CSTS since they are so dependent on volunteers and require expertise in such areas as workshop agenda development and facilitation. It is truly a synergistic relationship in which the results have been greater than what could have been achieved if the two groups had chosen to function separately. What allowed this unusual and difficult contracting mechanism to be effective? Several factors were identified that should be considered. One is the leadership in PVC/CS—the person with the vision and ability to allow the contractor the flexibility to conduct the project. There was sufficient oversight by PVC/CS so that it was kept informed of project operations—as can be seen in the dissatisfaction with progress during the first year and stepping in so that the necessary leadership change was made. Since then, the quarterly reports, meetings, and frequent phone contact seems to be sufficient to ensure that both parties know what has happened, what is happening, and what will happen in the immediate future. The Office of Procurement has been essential to the success of the CSTS in the last year. There are several unusual and possibly unique features about OP, which have apparently had a beneficial impact on the process. To begin with, the contract specialist has been involved with the CSGP for almost a decade. She had a good working relationship with the PVC/CS program staff and they worked to design something that they thought would improve both contractor performance, as well as PVO implementation of the child survival programs. She personally takes pleasure and satisfaction from working with the PVO community and enabling them to provide services to the needy target population in the developing world. She also takes part in project activities—she attended the last three CORE annual meetings. Recommendations ß PVC/CS should make every effort to maintain the current style of contract management focusing on openness, partnership, and flexibility to allow creativity on the part of the contractor to identify the optimal way to achieve results as specified in the contract. ß PVC/CS should facilitate (as raised in Section III.C) the continuation of the successful process by including the new chief of PVC/CS in the strategic planning exercise involving PVC/CS, CORE, and CSTS. C. BROADER PVC NEED It was evident to the evaluation team that there was tension between what CSTS was doing for PVC/CS and the broader needs of PVC. As someone said, CSTS is a victim of its own success. First, this is the only such technical support contract in the office. It is made possible because of the relatively generous child survival earmarked funds. PVC MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 42 has an administrative support contract that in the past has supplied mostly clerical assistance and support to all the units of the office, such as Matching Grant, Farmer-to￾Farmer, Food for Peace, and Child Survival. This contract was only recently awarded and will run for five years, but the evaluation team was unable to find out what is included in this iteration. The CSTS contract provides PVC/CS with services and support functions that other units like and want. These are generic and crosscutting activities, such as database development and maintenance, web site development and maintenance, and organizational capacity building. There has been the temptation for the office to ask the contractor to do similar tasks for the other units that it is doing for child survival. However, CSTS cannot accept additional work without existing activities being dropped. The CSTS project has no disposable time—all of its time is already booked, even before the program year begins. Assuming additional non-child survival tasks would affect the contractor’s performance in the CSTS contract and could make it difficult for them to perform acceptably in relation to their performance-based contract. Moreover, it is not within the scope of work. While PVC and the contractor have been vigilant in avoiding the temptation of expanding their mission, the problem still exists. There is a need for the leadership of the office to examine the needs of PVC across projects and identify generic support needs that cut across the various programs and sectors. To date, support has been provided in a unilateral, vertical manner to the individual PVC programs. Although CSTS is not an option for the support that PVC wants, there are contracting options available that they could utilize. One is the Management Organization and Business Improvement Services (MOBIS) contracting mechanism under the General Services Administration, Federal Supply Service. There are several different types of MOBIS including contracting services, facilitation services, survey services, training services, and support services. A number of contractors, including MACRO, are prequalified under these contracts and PVC might access the support it needs through this mechanism. This option has been explained to PVC, but it was reported that PVC has not begun to explore it. Recommendation ß The issue of broader/generic technical support of the other units in PVC should be addressed in a PVC strategic planning exercise for the future or additional contract (as included in recommendation 15 in Section III.C). 43 V. FINDINGS REGARDING PROJECT BUDGETING AND FINANCING Overall, the CSTS contract has been fiscally managed with care. The contract has utilized the resources in a balanced manner, with similar amounts being allocated to the three major recipients: PVOs, CORE, and PVC. The rate of project funds expended compared with obligated funds is very close to where it should be at this time in the project, based on experience to date and budget projections for the remainder of the contract. A. FISCAL RESPONSIBILITY Most of the individuals interviewed believed that CSTS was fiscally responsible, utilizing the resources available in a reasonable and prudent manner. The evaluation team noted the effective leveraging of funds and other forms of support from other sources that enabled the contractor to achieve more than they would have been able to with CSTS funding alone. For example, it brokered and was able to access considerable technical expertise from the Global Bureau and CAs to review and comment on relevant sections of the TRM. In addition, on a regular basis, CSTS has been able to refer PVOs with a special need to other agencies or groups that have specific, relevant expertise. Such referral precluded a lot of reinvention and duplication of effort while including a broad net of mutual technical support. Leveraging was also demonstrated in the development and presentation of the workshops, which was enhanced by the close working relationship with CORE. By sharing these costs, CSTS has been able to carry out a dozen training events at a reasonable cost per event. CSTS has also made efforts to improve the cost-effectiveness of its operation by reviewing its performance and considering how it could be improved in the future. For example, the second year CSTS provided logistical support for the child survival grant applications, it was able to reduce costs by one third by increasing efficiency. Another example of CSTS’s cost consciousness is the distribution of packets of materials at the semiannual meetings. This not only gives CSTS an opportunity to publicize and market the publications, but also saves on dissemination costs. A number of the recommendations raised as a result of this evaluation are not new. Some of them have been discussed and considered before and would possibly have been pursued and implemented had there been the funds to do so. Budgetary constraints are always a concern and a limiting factor. The evaluation team has made some suggestions and identified some options, but is not in a position to say what should or should not be done. As discussed in section III, strategic planning and priority setting will allow optimal results to be achieved from existing resources. For example, the cost of developing a core curriculum and training program is undoubtedly significant. However, if it were decided that this is the way that the clients want to proceed, then this becomes a MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 44 priority and several other activities, possibly some of the publications and one-time workshops, would be cut back. All the concerned parties need to agree on how the maximum benefit is derived from the resources available. This is the challenge that faces CSTS in the final two years of its contract and any support contract that follows. Recommendation ß The strategic planning effort (recommendation 13 in section III.C) will have to set priorities concerning how available resources will be utilized (i.e., if new activities are added, which activities will be reduced or eliminated). B. PROJECT EXPENDITURES Starting at the beginning of the second year, the CSTS contract has broken down its expenditures by 26 charge codes, which are divided into the following five major categories: ß tools/dissemination, ß CORE, ß PVO assistance, ß PVC support, and ß program guidelines. The coding does not allow for the contract to determine the percentage of its expenditures spent on each of the five contract results. However, it does allow for a breakdown by client. The figure below shows that there has been a balance in how the contract has spent project funds. PVC/CS support accounts for 25 percent of CSTS expenditures during the second year and the first part of the third year of the contract. CORE accounts for almost 20 percent of expenditures, while the PVOs account for about 18 percent. The other large category is General Contract Activities, which include such other direct costs as staff meetings, photocopying, and transport for project-wide purposes that cannot be attributed to the PVOs, CORE, or PVC categories. Figure 6: Project Expenditure by Category (for the period October 1, 1999—February 28, 2001) PVC 24.72% CORE 19.85% PVOs 17.98% Databases 9.93% JHU SOTA Papers 5.15% General Contract Activity 21.85% Customer Survey 0.35% Document Center 0.16% V. FINDINGS REGARDING PROJECT BUDGETING AND FINANCING 45 C. BUDGETED, PROJECTED, AND EXPENDED COSTS ANALYSIS The budget for the first three years (referred to as the base period) of the CSTS contract was $4,745,478. The first year was 16 percent under budget ($1,226,482 spent versus $1,454,883 budgeted) because of slow project start up and some of the problems that are discussed in section IV. This is not unusual in a new project, especially with a new contractor and a new contract mechanism. The second year was slightly (6 percent) over budget ($1,713,180 spent versus $1,610,199 budgeted) as the project was fully staffed and subcontract expenditures began to be incurred. Through five months of the third year, expenditures are running slightly ahead of the budget (about 3 percent) for the period ($730,478 spent versus $705,766 budgeted). The budgets for years 4 and 5 are $1,726,808 and $1,531,998, respectively, making a grand total of slightly over $8 million for the full five years. Salaries, indirect costs (fringe benefits, overhead, and general and administrative), and fees in the second year account for 85 percent of the expenditures, which is not unexpected in a technical assistance support contract that is labor intensive. More money has been spent in the Other Direct Cost category than originally budgeted—$65,000 was spent against less than $12,000 budgeted. The major Other Direct Costs expensed were reproduction (especially the printing of the KPC 2000+ notebook), meetings, and training sessions. The expenditure rate, as outlined above, is close to what it should be at this point. According to CSTS projections, it will have expended all of its obligated/budgeted funds by the end of September 2001 when the third year comes to an end. The projected budget does not include several activities that have been tentatively discussed, but are not included in the third year work plan (e.g., the Asia workshop on LQAS). If it is decided that these nonbudgeted activities are to take place, personnel salaries will have to be decreased, because they make up over 80 percent of the overall budget. According to CSTS leadership, this would be disruptive to the project but would not cause great personal hardship since MACRO would absorb the staff for the remainder of their time not used by CSTS. Some of the recommendations that are made as a result of this evaluation, such as training support, would require considerable resources—more than what exists in the contract during its last two years. An important issue to be considered in any strategic planning exercise (as discussed in section III.C) is the tradeoff between activities that are essential and those that are nonessential. Priorities must be established. Those activities with the most support and that are viewed as enabling the project to best achieve its objectives—and that are both feasible and affordable—should be strong candidates for funding. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 46 47 VI. CONCLUSION The CSTS project has performed satisfactorily over its first two and a half years and it should be continued for its fourth and fifth years. The project director who assumed the position in early 2000 has provided the leadership that has allowed CSTS to work effectively with PVC/CS and CORE to make progress in the five results specified in the performance work statement of the contract. The progress achieved to date has been the result of several factors. One is the effectiveness of USAID management. The CS division within PVC has focused its attention on the results, while being flexible in how it enforces the MAPS. In other words, it did not let MAPS obscure the results. Being one of the first performance-based contracts in a reengineered USAID and the first in BHR, CSTS was in some ways an experiment. Managed intelligently, in a results-oriented manner with a flexible and reasonable style, the project has made significant strides in achieving its goals. Another positive factor was effective leadership of CSTS by the contractor. The project director deserves credit for turning the project around after assuming command after the first year. His style was characterized by his ability and willingness to work in a collaborative, participatory, totally transparent manner with the most important partners, PVC/CS and CORE. The evaluation team found positive performance in all five results. Some of the most important achievements of the project to date include: ß a very positive focus on understanding and meeting client needs; ß frequent use of CSTS by PVO child survival grantees, especially its web site and response to requests for technical assistance; ß assistance to CORE working groups, especially in the planning, preparation, and logistical support of workshops; ß Bookmarks!, which is popular because it is brief and informs the PVO community about new and exciting developments in the child survival field; ß improved TRM, updated technical interventions, and more complete management/organizational development sections; ß revision of KPC into a new and improved tool—the KPC2000+; ß organizational development efforts, including the development of a new tool (i.e., ISA) for determining the capacity of health units; and, MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT 48 ß effective logistical support of PVC/CS, especially in the application and DIP review aspect. CSTS could be strengthened in the following ways: ß assume a more proactive approach to assist the smaller PVOs located outside the Washington, DC area; ß focus more capacity-building efforts on field offices and partner groups; ß explore the feasibility and cost of developing a distance learning strategy to effectively address the training needs of PVOs involved in CSGP; ß concentrate on the provision of brief updates on programmatically useful and innovative technical and management practices and interventions; ß translate more materials and publications to increase use by field offices and partner groups; ß provide assistance as appropriate to help forge a close relationship between CORE and its working groups and the child survival projects managed by CAs; and, ß conduct a joint strategic planning exercise with CSTS, PVC/CS, and CORE to determine the most appropriate way for CSGP objectives to be achieved in the future. With the positive progress made during the first half of the CSTS project and with momentum having been established, there is every reason to be confident that the results identified in the CSTS performance work statement will be achieved by September 2003 when the project draws to a close. A foundation has been laid which will allow exciting possibilities for growth and innovation. With the effective relationship between the three partner organizations, there is every reason to believe that positive results will continue to be achieved during the remainder of the CSTS project. ANNEXES A: Scope of Work ........................................................................................................ 51 B: Persons Contacted ................................................................................................... 57 C: Crosscutting Questions............................................................................................ 63 D: PVO Questionnaire ................................................................................................. 67 E: Miniquestionnaire ................................................................................................... 71 F: CSTS’ Publications and Trainings........................................................................... 73 G: References............................................................................................................... 75 H: BHR/PVC—CSTS Performance Work Statement.................................................... 81 I: Results, Indicators, and Performance Standards....................................................... 87 J: Evaluation Schedule ................................................................................................ 99 K: Current Child Survival PVO List........................................................................... 101 ANNEX A SCOPE OF WORK MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 52 Midterm Evaluation of the Child Survival Technical Support Project: Contract Number: FAO-C-98-00079-00 I. Title Activity: Midterm Evaluation of the Child Survival Technical Support Project Contractor conducting the Evaluation: Monitoring, Evaluation, and Design/Assessment Support (MEDS) HRN-I-99-000002-00. II. Objectives of the Evaluation 1. To assess the performance of the CSTS Project in order to: a) provide USAID/BHR/PVC with information on whether and/or how to continue funding for the first option year of the contract and b) to make suggestions for improving the work performance under the contract. 2. To assess the current performance based contract to determine whether or not the contract performance standards should be modified to better achieve the strategic objective of BHR/PVC. The primary audience of the evaluation will be USAID/BHR/PVC. In addition, the results will be shared with the CSTS Project and the CORE Group. III. Background THE OFFICE OF PRIVATE AND VOLUNTARY COOPERATION USAID’s Office of Private and Voluntary Cooperation, in the Bureau for Humanitarian Response (BHR/PVC), is the principal office in the Agency for articulating and promoting the development partnership with U.S. Private Voluntary Organizations (PVOs) and Cooperative Development Organizations (CDOs). The competitive grants programs funded and administered by BHR/PVC directly support U.S.-based PVOs and their in-country partners to address critical needs in developing countries and emerging democracies. These programs include: the Matching Grants Program, the Child Survival Grants Program, the Cooperative Development Program, the Farmer-to-Farmer Program, the Development Education Program, and the Ocean Freight Reimbursement Program. PVC’s strategic objective (SO), is “to increase the capability of BHR/PVC’s PVO partners to achieve sustainable service delivery.” THE CHILD SURVIVAL GRANT PROGRAM Since 1985, the Office of Private and Voluntary Cooperation, through the Child Survival Grants Program (CSGP), has invested Congressionally-directed Child Survival funds in community-based child survival programs implemented by U.S. PVOs and their local partners. The CSGP is intended to enhance the participation of PVOs in reducing infant, child, and maternal mortality in developing countries and to strengthen their ANNEXES 53 organizational, managerial, and technical competencies in these areas. The CSGP works with a broad range of organizations with a spectrum of different skills and abilities. Since the program began, PVC has funded more than 300 child survival programs in 44 countries, with more than 35 PVOs. The rigorous standards of the program challenge PVOs and their local partners to provide higher quality, sustainable, child survival interventions in a variety of program settings, from the smallest, most remote communities to large, district-wide programs, partnering with community groups to district health authorities. PVC strives for a participatory approach to this program, and continually elicits feedback from PVO partners to raise the standards of the program, to reflect the realities of the global development situation, and to incorporate the experience of the PVOs into the program guidelines. THE CHILD SURVIVAL TECHNICAL SUPPORT CONTRACT The purpose of the Child Survival Technical Support (CSTS) Project is to assist the Office of Private and Voluntary Cooperation to meet its strategic objective of strengthening the capability of PVOs to achieve sustainable service delivery in their child survival programs. It is a three-year contract with two option years. The start date was Sept. 98, and if the options are funded the contract will end Sept. 2003. The scope of the program is worldwide. Specifically, the contract aims to strengthen: ß the ability of PVO staff to design, manage and evaluate their child survival activities; ß the organizational capacity of PVO health units to fund, administer and provide backstopping support to a growing portfolio of child survival and health projects; ß the development and dissemination of information on the PVOs' successes and comparative advantage in implementing health programs; ß the ability of PVOs to strengthen host-country partners in long-term partnerships for child survival and health programming; and ß BHR/PVC's monitoring of performance, and management of the Child Survival Grants Program. To meet the above objectives, CSTS works with three main clients: BHR/PVC, individual PVOs, and the CORE Group (further explained below). Assistance to PVOs is provided through individualized technical assistance to PVOs in program design and implementation, and periodic state-of-the-art information on child survival interventions. CSTS also offers training opportunities focusing on measuring, documenting and disseminating results, and specialized workshops and conferences for personnel at the country, regional and headquarters level. For more information about CSTS, see their website at: http://www.childsurvival.com/. THE CORE GROUP Since the late 1980s, the PVOs funded under the CSGP have met on a regular basis to discuss technical needs. In 1997, the group formalized itself into a network known as The CORE Group (the Child Survival Collaborations and Resources Group). It now has more than 30 member organizations, all of which are current or past recipients of the Child Survival Grants Program. The goal of CORE is to assist member organizations to reduce MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 54 child and maternal mortality by improving health of under-served populations. CORE seeks to promote coordination and collaborations between the organizations as well as with outside agencies. CORE provides a networking function that facilitates exchange of learning between organizations. It also plays an advocacy role in raising public and donor awareness of child survival needs and PVO activities. For more information, see the CORE website http://www.coregroup.org/. IV. Tasks The Monitoring, Evaluation and Design/Assessment Support Project (MEDS) will provide the following technical and logistical support to complete a midterm evaluation of the CSTS Project through a “desk review” of project and other documents, and interviews with selected individuals. 1. Carry out necessary preparation activities for the mid-term evaluation, including but not limited to the following: a. Identify consultants b. Gather background materials for the team, including: · The CSTS Project contract · CORE Group documentation relevant to the evaluation · Key programmatic CSGP documents, including the RFA and Detailed Implementation Plan Guidelines · CSTS publications including quarterly reports, CSTS bookmarks, workshop reports, manuals, etc. · The CSTS website and data bases. 2. Organize a Team Planning Meeting with consultants and USAID. The purpose of the meeting is to: introduce consultants; provide a background briefing; produce a detailed workplan (which will indicate when necessary USAID approvals will be obtained); develop a draft outline of evaluation report; develop preliminary evaluation questions and tools; develop a contact list; develop a plan for stakeholder input; and determine how PVC will be kept informed of activities and make necessary approvals. 3. Hold two stakeholder meetings. The first will be to solicit stakeholder input into themes for the evaluation and the second will be to present preliminary findings to the same group. 4. Manage and advise the evaluation team. 5. Submit a final draft (four copies) of the report to PVC staff only. 6. Incorporate any necessary changes into the draft and submit a final version (6 copies) to PVC staff only. ANNEXES 55 V. Deliverables 1. A detailed workplan with deliverable dates (to be produced at the team planning meeting). 2. A data collection plan and data collection tools. 3. A fully edited, ready for distribution, mid-term evaluation document. The full contents of the document will be decided based on the team planning meeting and the stakeholder meeting. However, the following points must be covered in the document. · An analysis of performance under the contract · An assessment of the current performance standards and possible revision · Provide recommendations for or against continued funding VI. Team Composition Primary Team Members Team Leader Specialist in evaluation techniques. Experience in evaluation, preferably with experience evaluating PVO projects. Excellent written and oral skills. Knowledge of USAID and the PVO community. Preferably experience in PVO capacity building and organizational development. Child Survival Specialist Specialist in child survival with extensive experience implementing PVO programs. Experience in evaluation, including evaluating PVO projects. Excellent written and oral skills. Some knowledge of USAID. Additional Resources USAID Child Survival Specialists Two members of the G/PHN staff will participate in the team on a limited basis as time permits. They will participate in the team planning meeting, at which time their role and level of effort will be determined. MEDS MEDS will provide guidance to the team on capacity building/organizational development issues. Sandy Callier will facilitate the team planning meeting and be available as necessary to assist with the stakeholder meetings. VII. Evaluation Schedule January 2001 Agree on SOW, identify consultants, gather background materials February 2001 27 – 28 February Team Planning Meeting at the MEDS office MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 56 March 2001 1 March Team shares results of the Team Planning Meeting: workplan, preliminary table of contents 2 March Stakeholder meeting 5-15 March Team performs data collection and analysis 6 March Data collection plan and instruments shared with USAID 16-22 March Team writes draft report 23 March Team does reality check/debrief with USAID 26 March Stakeholder meeting to validate results/findings 30 March Final draft submitted to PVC April 2001 4 April USAID provides feedback on report to MEDS 6 April Final report submitted to USAID VIII. Relationships and Responsibilities In addition to providing consultants, MEDS will provide all technical administrative, logistical, and secretarial support required for completion of the Scope of Work. Technical directions from USAID will be as follows: Ann Hirschey 202 712 5734 ahirschey@usaid.gov Kate Jones 202 712 1444 kjones@usaid.gov Fax: 202 612 3041 Address: USAID/BHR/PVC Room 7.6 (7th floor) Washington, DC 20523 IX. Work week The contractor is authorized up to a five-day workweek with no premium pay. ANNEX B PERSONS CONTACTED MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 58 USAID, BHR/PVC Judy Gilmore Director Adele Liskov Deputy Director Peggy Meites Monitoring and Evaluation Specialist USAID, BHR/PVC/CS Katherine Jones Chief Ann Hirschey Project Officer Nitin Madav Technical Advisor Della Dash Technical Advisor USAID, BHR/FFP/DP Jean Capps Program Analyst USAID, BHR Contracts Office Raquel Powell Head USAID Ellen Wills Contract Officer USAID, G/PHN/HN Child Survival Elizabeth Fox CTO, CHANGE Project Nutrition and Maternal Health Eunyoung Chung CTO, FANta Project Kristen Marsh CTO, Linkages Project Child Survival Technical Support (CSTS) Leo Ryan Project Director Sandra Bertoli Monitoring and Evaluation Officer David Cantor Senior Management Information Specialist Molley Delaney Program Coordinator Michel Pacque Medical Advisor Eric Sarriot Community Sustainability Rikki Welch Dissemination Donna Espeut Research Analyst ANNEXES 59 MACRO International, Inc. Martin Vaessen Vice-President Johns Hopkins University Peter Winch Associate Professor Child Survival Collaboration and Resources Group (CORE) Victoria Graham Manager Joe Valadez Chairman of the Board David Newberry Board Member Larry Casazza Chair, IMCI Working Group Teresa Shaver Chair, Reproductive Health & Safe Motherhood Working Group Africare, Inc. Alan Alemian Director of Anglophone Region Sameh Saleeb Health Program Manager Malika Diara Health Advisor/Benin Aga Khan Foundation, USA Mona Khan Health Program Officer BASICS II Project Karen Leban PVO Liaison Officer CARE Sinjay Sinho Child Health Technical Specialist Catholic Relief Services Alphonso Rosales Technical Advisor for Health Environmental Health Project Massee Bateman Project Director MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 60 Esperança Reese Welsh Vice President of Programs FANta Project Bruce Cogill Project Director Foundation of Compassionate American Samaritans Amy Metzger Health Programs Director Freedom from Hunger Robb Davis Senior Technical Advisor, Maternal & Child Health Health Alliance International Mary Anne Mercer Deputy Director International Eye Foundation John M. Barrows Director of Programs International Rescue Committee Rick Brennan Director of the Health Unit La Leche League International Rebecca Magalhaes Director of Action & Development MAP International Marci Stoterau Health Programs Coordinator Medical Care Development Inc. Joe Carter Director Valasquez DeLavern Senior Project Officer Michael Hainsworth Senior Project Officer Mercy Corps Karla Pearcy Director of Health Programs ANNEXES 61 Partners for Development Dan Smith Public Health Technical Advisor Project Hope Betina Schwethelm Director, Maternal and Child Health Programs Quality Assurance Project Bart Burkhalter Evaluation Specialist Save the Children Eric Starbuck Child Survival Specialist Eric Swedberg Child Survival Specialist World Relief Melanie Morrow Child Survival Program Specialist Independent Consultants John Murray Gretchen Berggren MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 62 ANNEX C CROSSCUTTING QUESTIONS MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 64 I. Technical knowledge and expertise Where does it exist? Who tracks it? How is it kept up to date? How is the need for technical knowledge and expertise determined in the PVOs? Where is the expertise in the PVO community and are they open to sharing it? II. Dissemination: How is technical knowledge and expertise most effectively delivered and how useful has it been—one-on-one technical assistance, training, workshops, printed materials, web, tools, CD–ROMs, other media as developed? Are there other media and/or strategies that you are using? III. Quality Assurance: What efforts are being made by CSTS to determine quality and how are they being carried out? IV. Institutionalization: What efforts are being made at the PVO HQ, field and local partners to institutionalize technical knowledge and expertise? Are there ways that CSTS could support this? Does your PVO have a training specialist? RE: The training needs of consultants and new PVO staff vs. the old timers who keep pushing the bar higher. What efforts are being made by the PVOs to ensure that all PVO staff/consultants have a minimum level of technical expertise in child survival? What role has/should CSTS play(ed) in this? How has your PVO used the recommendations from the proposal review in the past to build its capacity? V. Use by the Field: How have field offices and local partners benefited from CSTS? Do field programs have access to the Internet and e-mail? Are CSTS materials shared with the field/local partners? Are HQ/field/partner staff attending CSTS trainings? Is field staff attending trainings? Can we interview your field staff? VI. Demand: Has CSTS approached you for input on topic selection? How are topics for technical assistance determined and prioritized? How do PVOs decide what is relevant? Which materials are getting used? ANNEXES 65 VII. Follow-up: Is follow-up being done with participants/clients to refresh, maintain and improve knowledge and skill level? VIII. Strategy: What is existing strategy for PVC, CORE, and CSTS? What do they want to achieve and how and who is going to do what? IX. Future Directions How could CSTS be strengthened and be more responsive to your PVO’s needs over the next two years? Any new areas or activities? X. What are the strengths and weaknesses of CSTS? XI. Have you had any positive or negative experiences with CSTS? XII. Changes in the longer-term future (i.e., next contract)? Utility and Quality of: Materials list Trainings list TA list Website usage/Consultants list PVO categories: Size: Length of time involved in BHR/PVC Child Survival: Amount of contact with CSTS: Proximity to Washington DC: Not currently funded: MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 66 ANNEX D PVO QUESTIONNAIRE MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 68 1. How long has your PVO been involved in the child survival program? 2. How many child survival projects do you currently have? 3. How many staff do you have backstopping your child survival projects at headquarters and regionally? 4. What are the capacity building needs of your current staff? headquarters/regional, field, partners, new hires (Have child survival experience?)? Are they different within these groups? 5. What are the challenges you face in trying to meet those needs? 6. How does your PVO come in contact with CSTS (working groups, CORE meetings, trainings, web site, list serves, receiving materials, technical assistance requests, requests from CSTS, others)? 7. Do you consider your PVO a frequent or infrequent user of CSTS services? Why? 8. Of the listed materials that you have seen, which have been the most/least useful? Why? Are they going to your field programs? 9. Of the listed trainings that your staff have attended which have been the most/least useful? Why? Who has been attending? ANNEXES 69 10. Have you used the CSTS web site? How? What has been useful/not useful? 11. Have you sought technical assistance from CSTS? What was the issue and response? 12. Do you feel that your PVO has had a voice in determining what topics CSTS is focusing on? How? 13. How could CSTS be strengthened and be more responsive to your PVO’s needs over the next two years? Any new areas or activities? 14. What are the strengths and weaknesses of CSTS? MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 70 ANNEX E MINIQUESTIONNAIRE MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 72 Greetings, We are writing to seek your input for the midterm evaluation of the Child Survival Technical Support (CSTS) Project by responding to the brief questionnaire at the end of this e-mail. (No need to go further if you have already, or will be, participating in the phone interviews.) To review, the purpose of the CSTS Project is to strengthen the capacity of BHR/PVC￾funded PVOs and their local partners to achieve sustainable child survival programs. This project has been implemented by Macro International since 1998 through a five￾year, performance-based contract, which requires a mid-term evaluation to assess: 1. The performance of the CSTS Project in order to: a) provide USAID/BHR/PVC with information on whether and/or how to continue funding for the first option year of the contract and b) to make suggestions for improving the work performance under the contract. 2. The current performance-based contract to determine whether or not the contract performance standards should be modified to better achieve the strategic objective of BHR/PVC. The Evaluation Team asks that you first review the attached, short document that outlines CSTS’s services and then respond to the three questions below. Responses can either be made by e-mail (cbillingsley@ltgassociates.org) or by phoning one of the Evaluation Team members, David Pyle or Garth Osborn, at (202) 898-0980 ext. 179. Because of time constraints, we would appreciate receiving your answers by Monday March 19, 2001 QUESTIONNAIRE: 1. What is your overall level of satisfaction with the assistance provided by CSTS to your PVO headquarters, your PVO field programs and/or the Working Groups? Please explain. 2. Describe your PVO’s needs for technical assistance for the next two years? 3. What topics and/or strategies should CSTS focus on over the next two years that would best meet your PVO’s needs? 4. Finally, we welcome any additional thoughts you have. Thank you for your time and input! David Pyle and Garth Osborn ANNEX F CSTS’ PUBLICATIONS AND TRAININGS MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 74 Materials Produced CSTS Bookmarks! CORE/CSTS Child Survival Connections A Manual of Manuals on Nutrition Interventions for PVOs (SOTA 1) Methodology and Sampling Issues for KPC Surveys (SOTA 2) Reaching Communities for Child Health and Nutrition: PVO Contributions to Community IMCI (SOTA 3) KPC 2000+ Rapid Catch—Core Assessment Tool for Child Health Institutional Self Assessment (ISA) Capacity Tool Bank USAID/BHR/PVC/ PVO Child Survival Grants Program Technical Reference Materials Participatory Program Evaluation Manual: Involving Program Stakeholders in the Evaluation Process Trainings Provided Regional Participatory Evaluation Workshop (Bolivia, July 1999) Regional Participatory Evaluation Workshop (Haiti, August 1999) Regional Participatory Evaluation Workshop (Senegal, May 2000) PVO Project Manager’s Toolkit Workshop (Harpers Ferry, WV, December 1999) CORE/CSTS HFA Workshop (Calverton, MD, January 1999) Sustainability Dialogue (Calverton, MD, March 2000) CSTS/CORE Web Workshop (December 2000) Effective Strategies to Promote Quality Maternal and Newborn Care (Nairobi, May 2000) Community-IMCI Workshop (Baltimore, MD, January 2001) M & E Update Meeting, featuring KPC 2000+ and LQAS TOT (Calverton, October 2000) CORE’s Malaria Update Meeting (Washington DC, September 2000) Quality Improvement: Practical Applications (June 1999) ANNEX G REFERENCES MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 76 Aubel, Judi. Participatory Program Evaluation Manual; Involving Program Stakeholders in the Evaluation Process. Second Edition. Burkhalter, Barton R. Cross-Cutting Review of CSV Mid-Term Evaluations. September 12, 1992. Burkhalter, Barton R. Concept Paper—Planning to Produce and Disseminate Information to PVOs on Child Survival through: State of the Art Papers, Best Practices and Lessons Learned. Prepared for Child Survival Technical Support Project. Macro International, Inc. February 1999. Child Survival Technical Support Project. An Assessment of the Technical Assistance Needs of PVC–funded Child Survival Projects and Their PVO Headquarters Offices. Calverton, Maryland. August 1999. ______. Brochure. ______. CSTS Collaboration Retreat; Summary and Outcomes, Annapolis Maryland, Calverton, Maryland, 17 February 1999. ______. Child Survival Grants Program Review 1998-1999. Calverton, Maryland: 25 September 2000. ______. Child Survival Project Manager’s Workshop; Managing for Results: Effectiveness, Innovation and Sustainability, Harper’s Ferry, West Virginia, 6-10 December 1999. ______. Child Survival Technical Support; 2000 Customer Satisfaction Survey. ORC Macro: 30 September 2000. ______. Child Survival Technical Support Project Retrospective Consultant Satisfaction Survey: Calverton, Maryland, August 1999. ______. Crucial Child Survival Interventions; A Checklist, Draft. ______. FY 2001 Workplan. ______. Evaluation Participative; Rapport d’Atelier, Dakar, Senegal, 8-13 May 2000. ______. Institutional Strengths Assessment Methodology; A tool for Assessing the Health Unit of a PVO; Self-Assessment Guide, September 2000. ______. Participatory Evaluation Workshops: Involving Program Stakeholders in the Evaluation Process, Reports of Trainings Held in La Paz, Bolivia, 26-30 July 1999 and 9-13 August 1999. ANNEXES 77 ______. Quarterly Report No. 1; 1 October 1998–31 December 1998. ______. Quarterly Report No. 2; 1 January 1999–31 March 1999. ______. Quarterly Report No. 3; 1 April 1999–30 June 1999. ______. Quarterly Report No. 4; 1 July 1999–30 September 1999. ______. Quarterly Report No. 1 (PROJECT YEAR 2); 1 October 1999–31 December 1999. ______. Quarterly Report No.2 (PROJECT YEAR 2); 1 January 200031–March 2000. ______. Quarterly Report No.3 (PROJECT YEAR 2); 1 April 2000 - 30 June 2000. ______. Quarterly Report No.4 (PROJECT YEAR 2); 1 July 2000 - 30 September 2000. ______. CSTS Request Log, 1 October 1998–7 March 2001. ______. R4 Submission to USAID/Bureau for Humanitarian Response/Office of Private and Voluntary Cooperation. ______. Sustainability Dialogue, Meeting Report 20 March 2000. April 2000. ______. Summary of CSTS Web Site and Online Information System. Child Survival Technical Support Project and the CORE Group. Child Survival Connections, Volume 1, Issue 1. ______. Child Survival Connections, Volume 1, Issue 2. ______. Health Facilities Assessment Tools Workshop: Final Report of the Monitoring and Evaluation Working Group, January 14–15, 1999. ______. Knowledge, Practices and Coverage Study, KPC 2000+. Calverton, Maryland: October 2000. Espeut, Donna. KPC 2000+ Field Guide (Draft). Child Survival Technical Support Project. ORC Macro. December 2000. Grobler-Tanner, Caroline. Documentation of Child Survival Collaborations and Resources; The CORE Network. CORE Network Study. February 2000. Macro International, Inc. Management, Organization and Business Improvement Services (MOBIS); Authorized Federal Supply Schedule Price List. Contract Number GS-23F- MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 78 9777H. Calverton, Maryland. Period Covered by Contract October 1, 1997–September 30, 2002. Pond, B. District Support for IMCI: Zambia. BASICS. July 1998. Putnam, E., et al. Final Evaluation of the Child Survival Support Program II. Health Technical Services Project. June 1997. The CORE Group. Child Survival CORE Group Working Groups Overview. ______. The CORE Group Action Plan—FY 1999. ______. The CORE Group Update, Volume II, June 1998. ______. Working Together in Health for Mothers, Children and Communities. United States Agency for International Development, Bureau for Humanitarian Response, Office of Private and Voluntary Cooperation. A Resource Guide for Private Voluntary Organizations; Working in Partnership with the US Agency for International Development. ______. An Analysis of Scores and Reviewer Comments from CSXVI Grant Applications. September 11, 2000. ______. BHR/PVC Child Survival Planning Calendar—FY2000. ______. Child Survival Grants Program Review: 1998–1999. September 25, 2000. ______. Child Survival Technical Support Project Request for Applications. Solicitation No. M/OP-01-002: FY 2001 September 1, 2000. ______. Contract between USAID and Macro International Inc. Contract No. FAO-C￾00-98-00079-00. September 30, 1998. ______. Contract between USAID and Macro International Inc. Contract No. FAO-C￾00-98-00079-00, Amendment/Modification No. 1. December 11, 1998. ______. Contract between USAID and Macro International Inc. Contract No. FAO-C￾00-98-00079-00, Amendment/Modification No. 2. July 30, 1998. ______. Contract between USAID and Macro International Inc. Contract No. FAO-C￾00-98-00079-00, Amendment/Modification No. 3. December 29, 1998. ______. Guide for Detailed Implementation Plans for PVO Child Survival Programs for FY 1999 (CS 15) Programs. December 1999. ANNEXES 79 ______. USAID Guidelines for Applying for Cost Extensions for Programs Ending in FY 2001. ______. PVO Child Survival Grants Program. Active Projects as of 3 January 2001. ______. USAID Office of Private and Voluntary Cooperation, Information Booklet. ______. PVO Child Survival Grants Program: Technical Reference Materials. Revised December 2000. Wagman, J. and Winch, P. Implementing and Evaluating Nutrition Interventions for Managers of PVO Child Survival Projects; A Guide to Manuals, Guidebooks and Reports. Winch, P., et al. Reaching Communities for Child Health and Nutrition: NGO Contributions to Community IMCI, Draft for Comment. November 2000. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 80 ANNEX H BHR/PVC CSTS PERFORMANCE WORK STATEMENT MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 82 RESULT 1: CS Technical and Management Expertise The Contractor shall be responsible for measurable improvement in the knowledge and skills of PVO headquarters and field staff of CS PVO programs currently funded under the CSAP and, to a limited extent, other USAID sources. Indicators: · Up-to-date assessment of CSAP project staff skills and needs in program management and quality, and the critical child survival interventions and approaches. · Timely, high quality short and long term technical assistance and training provided by contractor staff or other technical experts hired with contract funds. · Technical assistance and training from this and other USAID-funded sources must complement each other. · An up-to-date database of qualified independent (US and FN) consultants in all major child survival interventions, and in all relevant project management categories, available to all PVOs in PVC’s Child Survival grants program, from which they may choose to identify and directly hire with their own sources technical specialists to meet their projects’ technical assistance needs. Minimum Acceptable Performance Standards: a) beginning one year from the date of this contract until the end of contract period, there shall be no unmet requests reported by PVO CSAP project/s for the CS technical expertise covered in the contract; b) beginning one year from the date of this contract until the end of contract period, at least 85 percent of CSAP PVO staff who receive technical assistance and/or training from the contractor shall show an increase in technical and management skills for achieving their child survival project objectives; c) beginning one year from the date of this contract until the end of contract period, at least 85 percent of CSAP PVO staff who have received technical assistance from the contractor report that the quality of the technical assistance was good or better; d) within nine (9) months of the date of this contract, all PVOs in the child survival grants program shall have knowledge of the existence of this database; and e) beginning one year from the date of this contract until the end of contract period, at least 85 percent of CSAP project managers who have hired consultants from ANNEXES 83 the CSTS database report that they are highly satisfied with the quality and timeliness of the technical assistance received over life of the contract. RESULT 2: Technical Child Survival Information The Contractor shall be responsible for creating and operating an information dissemination system with the following minimum features: · Accessible by PVO HQ and CSAP project staff to the latest developments related to the state-of-the-art in child survival programming; · Accessible by PVO HQ and CSAP project staff to published and unpublished materials on “lessons learned” and “best practices” in CS programming from grantees and others; and · Capable of handling the rapid exchange of PVO-identified information and PVO￾generated requests for information on CS issues. Indicators: · PVO knowledge of the information dissemination system · Currentness of information disseminated · Utilization of the information system · Level of satisfaction with information disseminated · CSAP best practices identified and disseminated Minimum Acceptable Performance Standards: a) Within nine (9) months of the date of this contract, all CSAP PVO project officers shall have knowledge of the information dissemination system; b) Within six (6) months of the date of this contract and at a minimum of on a semi￾annual basis thereafter, all CSAP project officers shall receive contractor compiled information on child survival State of the Art (SOTA), Lessons Learned (LL), and Best Practices (BP); c) By end of contract period, CSAP project officers from at least 75 percent of all PVOs in the CSAP (1998-2002) shall have used this system to ask for information on CS issues; d) Beginning one year from the date of this contract until the end of contract period, at least 85 percent of CSAP project officers shall report that they are highly satisfied with the content, quality and timeliness of the information received via this mechanism; e) Three (3) CSAP “best practices” disseminated to all PVOs in CSAP annually. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 84 RESULT 3: Project Management Tools and Skills The Contractor shall be responsible for measurable improvement in CSAP staff skills in collecting, analyzing, and presenting child survival project information. Activities may include provision of technical assistance, training, networking or mentoring services. Indicators: · Appropriate baseline and end-of-project surveys conducted for all project in CSAP · Knowledge level of CSAP participants on quantitative research methodology · Knowledge level of CSAP staff on qualitative research methodology · PVOs accurately measure and report on program results · PVO and NGO staff trained in project management methods and tools Minimum Acceptable Performance Standards: a) beginning fifteen (15) months from the date of this contract, all CSAP PVOs measure CS programs at the baseline and at end-of-project using: measurable, quantifiable results oriented objectives; valid, objective, reliable, practical indicators; and scientific sampling techniques and correctly validate analyze, compare, and interpret baseline and final results; b) beginning six (6) months from the date of this contract, all CSAP headquarters staff and field staff responsible for new CSAP project use situation appropriate, methodologically sound, qualitative and qualitative research techniques for program planning, monitoring and evaluation as presendeid in their DIPs; c) beginning fifteen (15) months from the date of this contract all CSAP headquarters project officers shall be able to articulate their “program results” using a complete array of monitoring and evaluation tools (including community level assessment information, health facility delivery statistics and project management indicators and benchmarks). CS grantees/recipients shall be able to uses multiple indicators to triangulate actual trends and achievements and describe results in final program reports; d) by end of contract period, all CSAP PVOs have at least two staff members (2 headquarters or 1 headquarters and 1 field) trained in project management methods (including project design and work plan development and planning and conducting evaluations); e) all CSTS trained trainers develop training plans and conduct at a minimum 2 trainings in program management methods and tools; ANNEXES 85 f) beginning 18 months from date of the contract and annually thereafter, at least 85 percent of PVO staff express satisfaction with the effectiveness of the managmenet tools and methods and their ease of use at the project implementation site. RESULT 4: Organizational Development The Contractor shall be responsible for measurable improvement in the PVOs health units’ ability to (a) plan and direct their health portfolios including the development of the skills of health unit headquarters and field staff in such areas as negotiation, conflict resolution, human resource development, team building, leadership, strategic planning, financial and administrative management, sustainability and manage of donor and constituency relations and (b) develop the above mentioned skills with NGOs in developing countries. Indicators: · Annual assessment of CSAP health units institutional strength using PVC accepted tools · Knowledge level of CSAP health unit staff of relevant organizational development, strategies and tools · Knowledge level of CSAP health unit staff of sustainability strategies · PVO training and technical assistance visits to local partner/NGO in institutional strengthening. Minimum Acceptable Performance Standards: a) An assessment of the capacity of PVOs to manage their health portfolio shall be completed within one (1) year form beginning date of this contract; b) An institutional strength assessment methodology (including institutional strength indicators) developed/adapted from existing tools and submitted to COTR within six (6) months from the beginning date of this contract; c) Beginning 18 months from the date of this contract, at least 85 percent of staff from each CSAP PVO shall have been trained (or shall have received technical assistance) in organizational development subject matter; d) By end of contract period at least 85 percent of CSAP funded PVOs shall have provided technical assistance to 50 percent of their partners NGOs in institutional strengthening activities; e) Beginning 18 months from the date of this contract, at least 75 percent of PVOs previously funded under the CSAP shall demonstrate knowledge of appropriate sustainability strategies as presented in their new applications. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 86 RESULT 5: CSAP Management by PVC The Contractor shall assist BHR/PVC in improving their monitoring capability of the CSAP. (Refer to Section III: Crosscutting Issues). Indicators: · Operational BHR/PVC CSAP health and management information system · Timely hiring of external reviewers for grant applications and documents to meet BHR/PVC deadlines · Timely response to PVC requests for data and information on CSAP · Annual summary analysis of CSAP performance Minimum Acceptable Performance Standards: a) Within twelve (12) months from the date of this contract, CSTS contractor shall create and operate a Health and Management Information system including at a minimum the following elements: ß a project database that at a minimum: includes technical assistance to projects and costs by months and work category; identifies target achievements; specifies total project costs’ and assigns a unique identification number to each project in order to allow relevant databases to be linked and shared. ß PVO Institutional Strength Indicators Database that at a minimum tracks progress in the capacity of CSAP grantees to develop and manage child survival health programs. ß Child Survival Indicators Database accessible to USAID and PVOS that at a minimum contains the necessary statistical information to determine the significance of the results of the CSAP programs. ß Others as negotiated by USAID-BHR/PVC. b) Within two (2) months from the date of this contract, CSTS contractor shall develop and implement a strategy to assist with the CSAP application and DIP reviews and program evaluations. The plan, at a minimum, will hire technical experts as reviewers and administrative support personnel as recorders for all meetings. c) Within two (2) months from the date of this contract, CSTS contractor shall demonstrate an ability to respond within a maximum of 2 working day to ad hoc requests from BHR/PVC CSAP for data and CSAP information (These requests are usually in response from Congress, the White House or the USAID Administrator); d) Summary analyses of the performance of the CSAP conducted by December 31 for each calendar year (1998–2001), and submitted to the COTR by the following February 1. ANNEX I RESULTS, INDICATORS, AND PERFORMANCE STANDARDS COMMENTS AND RECOMMENDATIONS MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 88 RESULT 1: CS Technical and Management Expertise: The Contractor shall be responsible for measurable improvement in the knowledge and skills of PVO headquarters and field staff of CS PVO programs currently funded under the CSAP and, to a limited extent, other USAID sources. Indicators: · Up-to-date assessment of CSAP project staff skills and needs in program management and quality, and the critical child survival interventions and approaches. · Timely, high quality short and long term technical assistance and training provided by contractor staff or other technical experts hired with contract funds. · An up-to-date database of qualified independent (US and FN) consultants in all major child survival interventions, and in all relevant project management categories, available to all PVOs in PVC’s Child Survival Grants program, from which they may choose to identify and directly hire with their own sources technical specialists to meet their projects’ TA needs. Performance Standards Status9 Comments/Recommendations i. Beginning one year from the date of this contract until the end of contract period, there shall be no unmet requests reported by PVO CSAP project/s for the CS technical expertise covered in the contract. This is being met. Recommendation: No change. The need to focus technical assistance on those PVOs with the greatest needs is further discussed in section III.C. ii. Beginning one year from the date of this contract until the end of contract period, at least 85percent of CSAP PVO staff who receive technical assistance and/or training from the contractor shall show an increase in technical and management skills for achieving their child survival project objectives. This is being measured by “perceived applicability” of tools and skills which was assessed through a conference evaluation at the CSTS project manager’s workshop in December 1999. No other strategy has been adopted to measure this Standard since. This performance standard is not being measured due to the inherent difficulty in drawing a causal connection between the technical assistance and training provided by CSTS and change in the technical and management skills of CSAP PVO staff. It is also questionable how this standard could be directly measured other than by self-perception of increased skills, which is highly biased. Change in overall PVO capacity resulting from technical assistance and the tools is already being measured through performance standards 3 i-iii that rely on the assessment of PVO applications, midterm/final evaluations, and DIPs. Recommendation: Replace or modify this performance standard to assess how the PVOs are using the tools and/or knowledge gained to impact their projects. 9 The status has been taken from the most recent CSTS quarterly report for October—December 2000. ANNEXES 89 iii. Beginning one year from the date of this contract until the end of contract period, at least 85percent of CSAP PVO staff who have received technical assistance from the contractor report that the quality of the technical assistance was good or better. 87.2 percent of technical assistance recipients reported that the quality of technical assistance was good or better based on the 2000 customer satisfaction survey. Recommendation: No change. iv. Within nine (9) months of the date of this contract, all PVOs in the child survival grants program shall have knowledge of the existence of this database. All child survival PVOs have been informed of the existence of the consultant database. Customer satisfaction survey indicates 92 percent of respondents are aware of the database and 52 percent of those have accessed it. The contract discusses four different databases (consultant, CS project, institutional strength indicators, and CS indicators). To avoid confusion, it would be useful to clarify that this standard refers to the consultant database. Recommendation: Reword to clarify that this standard refers to the consultant database. v. Beginning one year from the date of this contract until the end of contract period, at least 85 percent of CSAP project managers who have hired consultants from the CSTS database report that they are highly satisfied with the quality and timeliness of the technical assistance received over life of the contract. 83 percent of respondents to customer satisfaction survey report having hired a consultant from the database. 93 percent of these respondents were satisfied with the database. PVO satisfaction with the consultants hired from the database is not being measured. As discussed above in result 1 findings, the consultant database facilitates reference checks of potential consultants, without having to deal with the liability issues of having written evaluations posted on the website. Further, the PVOs noted areas where the quality of the consultant database could be strengthened. Recommendation: Further options should be explored that would provide a mechanism for ensuring that the consultants in the database are sufficiently skilled and qualified. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 90 RESULT 2: Technical Child Survival Information: The Contractor shall be responsible for creating and operating an information dissemination system with the following minimum features: · Accessible by PVO headquarter and CSAP project staff to the latest developments related to the state-of-the-art in child survival programming; · Accessible by PVO headquarter and CSAP project staff to published and unpublished materials on “lessons learned” and “best practices” in CS programming from grantees and others; and · Capable of handling the rapid exchange of PVO-identified information and PVO-generated requests for information on CS issues. Indicators: · PVO knowledge of the information dissemination system. · Currentness of information disseminated. · Utilization of the information system. · Level of satisfaction with information disseminated. · CSAP best practices identified and disseminated. Performance Standards Status Comments/Recommendations i. Within nine (9) months of the date of this contract, all CSAP PVO project officers shall have knowledge of the information dissemination system. Customer Satisfaction Survey demonstrates that all respondents are familiar with the CSTS information dissemination system. Recommendation: No change. ii. Within six (6) months of the date of this contract and at a minimum of a semi-annual basis thereafter, all CSAP project officers shall receive contractor-compiled information on child survival State of the Art (SOTA), Lessons Learned (LL), and Best Practices (BP). As of December 31, 2000, CSTS has disseminated 65 Bookmarks!, 2 SOTA papers, 2 CS Connections, and has posted 18 success stories on its website. Recommendation: No change. iii. By end of contract period, CSAP project officers from at least 75 percent of all PVOs in the CSAP (1998-2002) shall have used this system to ask for information on CS issues. Per CSTS request log, 92 percent of PVOs funded in the CSGP (n=26) have requested information or assistance from CSTS. As discussed in Section III.C., the PVOs that are not seeking technical assistance are most often the smallest, newest, and farthest away from Washington, DC. A more proactive approach is recommended in reaching out to these PVOs, as they are often the ones with the greatest needs for technical assistance. Recommendation: The level of the target should be increased to 100 percent, based on the need for CSTS to proactively develop and maintain relationships with all the PVOs. Reference discussion in Section III.C. ANNEXES 91 iv. Beginning one year from the date of this contract until the end of contract period, at least 85 percent of CSAP project officers shall report that they are highly satisfied with the content, quality and timeliness of the information received via this mechanism. As reported in the Customer Satisfaction Report: ß 94 percent satisfaction with content of the CSTS technical assistance. ß 91 percent satisfaction with the quality of the technical assistance. ß 91 percent satisfaction with the timeliness of the response. Recommendation: No change. v. Three (3) CSAP “best practices” disseminated to all PVOs in CSAP annually. See performance standard 2 ii above. This performance standard is redundant with 2 ii above. Recommendation: This standard could be combined with 2 ii above. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 92 RESULT 3: Project Management Tools and Skills: The Contractor shall be responsible for measurable improvement in CSAP staff skills in collecting, analyzing, and presenting child survival project information. Activities may include provision of technical assistance, training, networking or mentoring services. Indicators: · Appropriate baseline and end-of-project surveys conducted for all projects in CSAP. · Knowledge level of CSAP participants on quantitative research methodology. · Knowledge level of CSAP participants on qualitative research methodology. · PVOs accurately measure and report on program results. · PVO and NGO staff trained in project management methods and tools. Performance Standards Status Comments/Recommendations i. Beginning fifteen (15) months from the date of this contract, all CSAP PVOs measure CS programs at the baseline and at end-of-project using: measurable, quantifiable, results oriented objectives; valid, objective, reliable, practical indicators; and scientific sampling techniques and correctly validate analyze, compare, and interpret baseline and final results. Analysis of CS X and CS XI finals, as well as CS XIII midterms suggests that all funded PVOs are taking measurements at baseline and final in manners that are acceptable with regards to this result. Recommendation: No change. ii. Beginning six (6) months from the date of this contract, all CSAP headquarters staff and field staff responsible for new CSAP projects use situation appropriate, methodologically sound, qualitative and quantitative research techniques for program planning, monitoring and evaluation as presented in their DIPs. Review of CS XIV DIPs revealed that program planning and monitoring techniques are not described in sufficient detail to accurately assess this result. As currently worded, this performance standard is focused on individuals in the PVOs rather than the PVO overall. However, it is being tracked based on PVO performance which is reasonable considering the need to not only train individuals but build PVO capacity. Further, while there are several places in the DIP guidelines where the PVOs can discuss how they are using their research results for program planning, monitoring, and evaluation, there is not a specific question that addresses these issues. Recommendation: Focus this performance standard at the PVO level rather than the individual staff level Recommendation: Reassess/amend the DIP guidelines to ensure that the PVOs are including information in their DIPS that can be used to assess this performance standard. ANNEXES 93 iii. Beginning fifteen (15) months from the date of this contract all CSAP headquarters project officers shall be able to articulate their “program results” using a complete array of monitoring and evaluation tools (including community level assessment information, health facility delivery statistics and project management indicators and benchmarks). CS grantees/recipients shall be able to use multiple indicators to triangulate actual trends and achievements and describe results in final program reports. As of January 2000, analysis of the final evaluations indicates that this isn’t happening yet. Findings from the analysis will be used as a baseline for developing project activities that lead more directly to this result. PVC/CS has noted that 1998 should be the baseline, before CSTS initiated the project rather than 2000. However, CSTS never completed the January to March 1999 analysis. Further, the first half of this standard appears to be redundant with performance standard 3c above. Recommendation: This Performance Standard should be based on each PVO rather than each individual staff due to the high staff turnover rate and the need for institutionalizing this capacity within the PVOs. Recommendation: Consideration should be given to dropping the first half of this performance standard: “Beginning fifteen (15) months from the date of this contract all CSAP grantees/recipients shall consider multiple indicators to triangulate actual trends and achievements and describe results in final program reports.” iv. By end of contract period, all CSAP PVOs have at least two staff members (2 headquarters or 1 headquarters and 1 field) trained in project management methods (including project design and work plan development and planning and conducting evaluations). 61.5 percent (n=26) of all CSGP PVOs presently have two staff members trained in various management methods. 88.4 percent of all CSGP PVOs have at least one member trained. Recommendation: Use of the term “have” is ambiguous and implies to possess. Because of the high staff turnover in the PVOs, it would be appropriate to change this performance standard to: “ By end of contract period, all CSAP PVOs will have had…” (See discussion in section III.C.) MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 94 v. All CSTS trained trainers develop training plans and conduct at a minimum 2 trainings in program management methods and tools. 19 of 29 monitoring and evaluation update participants developed plans to apply their learning. While it is a reasonable expectation for workshop attendees to develop a plan to pass on what they learned to their colleagues at headquarters and the field, it is not clear to what extent it is really happening or the impact it is having at the field level through this monitoring method. Recommendation: The need to monitor and follow up on what training content is reaching the field programs should be explored further because it is such an important part of this project. This discussion is continued in section III.C: Training/Capacity Building. vi. Beginning 18 months from date of the contract and annually thereafter, at least 85 percent of PVO staff express satisfaction with the effectiveness of the management tools and methods and their ease of use at the project implementation site. Customer satisfaction study indicated that 82 percent of respondents reported some increase in management and technical skills as a result of CSTS technical assistance. Recommendation: No change. ANNEXES 95 RESULT 4: Organizational Development: The Contractor shall be responsible for measurable improvement in the PVOs health units’ ability to (a) plan and direct their health portfolios including the development of the skills of health unit headquarters and field staff in such areas as negotiation, conflict resolution, human resource development, team building, leadership, strategic planning, financial and administrative management, sustainability and management of donor and constituency relations, and (b) develop the above mentioned skills with NGOs in developing countries. Indicators: · Annual assessment of CSAP health units institutional strength using PVC accepted tools. · Knowledge level of CSAP health unit staff of relevant organizational development, strategies, and tools. · Knowledge level of CSAP health unit staff of sustainability strategies. · PVO training and technical assistance visits to local partner/NGO in institutional strengthening. Performance Standard Status Comments/Recommendations i. An assessment of the capacity of PVOs to manage their health portfolio shall be completed within one (1) year from beginning date of this contract. One pilot assessment completed. 3 PVOs presently in the prep phase of their assessments. 4 additional PVOs negotiating participation in the ISA. ß As stated in section III.B. on result 4 findings, the development and implementation of the ISA has taken longer than originally envisioned. ß An Assessment of the Technical Assistance Needs of PVC–funded Child Survival Projects and Their PVO HQ Offices was completed in August 1999. ß There is confusion between this performance standard, the related indicator above, and the first indicator in result 1, which is also related. Recommendation: CSTS and PVC/CS should clarify what is entailed in meeting this performance standard and the two above-mentioned indicators. ii. An institutional strength assessment methodology (including institutional strength indicators) developed/adapted from existing tools and submitted to the CTO within six (6) months from the beginning date of this contract. ISA methodology developed as of September 2000. Recommendation: No change. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 96 iii. Beginning 18 months from the date of this contract, at least 85 percent of staff from each CSAP PVO shall have been trained (or shall have received technical assistance) in organizational development subject matter. Based on workshop attendance records, the request log, and field visits, 96 percent of PVOs with an active grant have received OD￾related training or technical assistance. The target population is not clearly defined, nor is it consistent with the way this performance standard is being measured (by PVO versus by staff person). Further, the deadline is not realistic in the context of the participatory approach undertaken by the project in the design of the tools. Recommendation: Change this performance standard to read: “ By month 36, each PVO health unit will have received training or technical assistance in organizational development subject matter.” iv. By end of contract period at least 85 percent of CSAP funded PVOs shall have provided technical assistance to 50 percent of their partners NGOs in institutional strengthening activities. CS X an CS XI final evaluations indicate that 100 percent (n=5) and 94 percent (n=18) respectively, of PVOs in these cohorts provided institutional strengthening technical assistance to their local partners. Recommendation: No change. v. Beginning 18 months from the date of this contract, at least 75 percent of PVOs previously funded under the CSAP shall demonstrate knowledge of appropriate sustainability strategies as presented in their new applications. Analysis of application scores indicates that 82.1 percent of PVOs previously funded under the CSGP demonstrate knowledge of appropriate sustainability strategies (based on scores of 70 or above on the sustainability section.) Recommendation: No change. ANNEXES 97 RESULT 5: CSAP Management by PVC: The Contractor shall assist BHR/PVC in improving their monitoring capability of the CSAP. Indicators: · Operational BHR/PVC CSAP health and management information system. · Timely hiring of external reviewers for grant applications and documents to meet BHR/PVC deadlines. · Timely response to PVC requests for data and information on the CSAP. · Annual summary analysis of CSAP performance. Performance Standards Status COMMENTS/RECOMMENDATIONS i. Within twelve (12) months from the date of this contract, CSTS contractor shall create and operate a health and management information system including at a minimum the following elements: ß a project database that at a minimum: includes technical assistance to projects and costs by months and work category; identifies target achievements; specifies total project costs’, and assigns a unique identification number to each project in order to allow relevant databases to be linked and shared. (Continued on next page.) ß Project database has been established and is functional. Recommendation: No change. ß PVO institutional strength indicators database that at a minimum tracks progress in the capacity of CSAP grantees to develop and manage child survival health programs. ß Child survival indicators database accessible to USAID and PVOs that at a minimum contains the necessary statistical information to determine the significance of the results of the CSAP programs. ß Others as negotiated by USAID￾BHR/PVC. ß The PVO ISA Database is in development parallel to result 4. ß CS indicators database is under development until rapid catch data is collected. Recommendation: No change. Recommendation: No change. MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 98 ii. Within two (2) months from the date of this contract, CSTS contractor shall develop and implement a strategy to assist with the CSAP application and DIP reviews and program evaluations. The plan, at a minimum, will hire technical experts as reviewers and administrative support personnel as recorders for all meetings. This standard is being met. Recommendation: No change. iii. Within two (2) months from the date of this contract, CSTS contractor shall demonstrate an ability to respond within a maximum of 2 working days to ad hoc requests from BHR/PVC CSAP for data and CSAP information (These requests are usually in response from Congress, the White House, or the USAID Administrator). This standard is being met. This has sometimes taken longer than two days to prepare. Recommendation: Change deadline to: “within timeframe jointly agreed to by CSTS and PVC/CS.” iv. Summary analyses of the performance of the CSAP conducted by December 31 for each calendar year (1998-2001), and submitted to the CTO by the following February 1. Summary analysis completed for 1998 to 1999, dated October 2000. 1999 to 2000 has not been completed yet. However, CSTS did provide information in February 2001 sufficient for the R4. Recommendation: No change. ANNEX J EVALUATION SCHEDULE MIDTERM EVALUATION OF THE CHILD SURVIVAL TECHNICAL SUPPORT PROJECT 100 Date Activity 26 February Background and document review 27-28 February Team planning meetings 1 March Development of work plan & draft table of contents 2 March 1 st stakeholders’ meeting 5 March Retooling of work plan and draft table of contents 5-8 March Development of assessment tools; questionnaire, mini-questionnaire, interview list and interview questionnaire 9-15 March Data collection and analysis 16 March Meeting with Kate Jones & Ann Hirschey, USAID/BHR 16-22 March Draft report writing 21 March Meeting with Leo Ryan, CSTS 23 March Briefing on results to USAID/BHR team 26 March Briefing on results to CSTS team Briefing on results to 2nd stakeholders’ meeting 27-30 March Report writing and compilation Report delivered to USAID/BHR 2-4 April USAID/BHR reviews and comments on report. 5-6 April Report is finalized. 6 April Final draft report delivered to USAID/BHR. 20 April Final, published report delivered to USAID/BHR. ANNEX K CURRENT CHILD SURVIVAL PVO LIST