Evaluation of USAID-Funded Programs to Train International Tuberculosis Control Consultants September 15, 2005 1800 K Street NW Washington, DC 20006 USA Tel: 202.822.0033 Fax: 202.457.1466 www.path.org This evaluation was made possible through support provided by the Office of Procurement, Bureau for Global Health, U.S. Agency for International Development, under the terms of Contract No. GHS-I-00-03-00034-00. The opinions expressed herein are those of the author(s) and do not necessarily reflect the views of the U.S. Agency for International Development. ii Acknowledgements The consultant team would like to express appreciation to the many tuberculosis experts in the world committed to providing national programs with the high quality technical assistance they need to save lives. It is rare to have the opportunity to speak with people from all corners of the globe who have the passion and dedication of the “TB community.” We would like to thank the trainees, mentors, supervisors, and clients we spoke with as part of this evaluation for their time and thoughtful contributions. Included in this community is the USAID team that is seeking ways to invest funds to increase the number and quality of consultants available to support TB programs. In particular we would like to thank Cheri Vincent, Susan Bacheller, and Peg Marshall of USAID. iii TABLE OF CONTENTS EXECUTIVE SUMMARY ......................................................................................................... V I. INTRODUCTION..................................................................................................................... 1 II. DESCRIPTION OF SCOPE OF WORK.............................................................................. 3 III. METHODOLOGY................................................................................................................. 4 IV. OVERVIEW OF CONSULTANT TRAINING PROGRAMS AND FUNDING ............. 5 V. ASSESSMENT OF APPRENTICESHIP PROGRAMS AND SHORT COURSES TO TRAIN INTERNATIONAL CONSULTANTS ....................................................................... 13 VI. RECOMMENDATIONS ..................................................................................................... 26 ATTACHMENT I: SCOPE OF WORK .................................................................................. 37 ANNEX 1 ..................................................................................................................................... 44 ANNEX 2 ..................................................................................................................................... 45 ATTACHMENT II: DOCUMENTS CONSULTED............................................................... 46 ATTACHMENT III: INTERVIEW INSTRUMENTS ........................................................... 49 iv Executive Summary The number of skilled consultants needed to provide technical assistance to national tuberculosis (TB) programs has outstripped the supply in recent years. This increased need for technical assistance is driven by a combination of factors that include increased sources and levels of funding, increasing complexity of the disease (TB/HIV), and growing recognition that TB programs need to be integrated into the broader health system. With these trends comes the need for both more generalists who can take a holistic view of national challenges and specialists who can provide targeted assistance in key areas. Recognition of this shortage of skilled human resources prompted USAID to support a range of programs between 2003 and 2005 to train international consultants. The purpose of this evaluation is to assess the impact of USAID investments to expand the availability of qualified technical assistance and to provide recommendations to guide future USAID investments to expand the availability of consultants that both build on existing programs and develop possible new strategies. In the past three years, USAID has financed both short courses and long-term apprenticeship programs to train international TB consultants. This evaluation looked at four apprenticeship programs: one managed by the Pan American Health Organization (PAHO) and funded by a USAID grant and three managed by partners of the Tuberculosis Coalition for Technical Assistance (TBCTA) —, KNCV Tuberculosis Fund (KNCV), The International Union Against TB and Lung Disease (“The Union” or IUATLD), and the World Health Organization (WHO), and funded by USAID through TBCTA. We also examined the short course to train international TB consultants run in Sondalo, Italy, and funded by USAID through TBCTA. Limited information was available to evaluate the short course run by the American Thoracic Society (ATS) and funded by TBCTA for North American medical practitioners. In addition, we looked at a web-based course to train TB consultants, run by the University of Alberta that was oriented to North Americans in the test phase. We looked at written documentation, data about participants and costs, and course materials, and conducted over 30 interviews with trainees, mentors, supervisors, and clients. The ultimate measure of success of these programs is whether they offer a cost-effective and expedient way to increase the number of skilled consultants so that more lives are saved. Of the nine people who have completed long-term apprenticeship programs, five are now working as full-time international TB consultants. While it is difficult to reach definitive conclusions about the cost-effectiveness of long-term programs, given these small numbers, USAID has spent an average of $158,900 to produce each successfully engaged international consultant. In comparison, of the 34 respondents to a survey from the Sondalo course (out of 55 total participants), 27 replied that they have worked as international consultants since the course ended and 21 reported having done an international consultancy for the first time. It is difficult, of course, to fully attribute work as an international consultant to this course, but given very rough assumptions we can estimate that it cost USAID $11,178 to develop a person who worked as an international consultant after their training experience. As the number of people graduated from long-term programs is small (9 out of 17 have completed training) and limited follow-up information is available from participants of short courses, we considered multiple sources of information to assess the success of each approach. In v addition to reporting on what trainees are doing for work after completing training, we analyzed the quality of training approaches and how they were implemented, compared each approach to sound practices known to effectively train adults, and interviewed a broad range of stakeholders to understand perceptions of what worked well and what could be improved. Our assessment of apprenticeship programs indicates that one of the key contributors to success is a capable and committed mentor. Another important feature that contributed to success was a comprehensive assessment of skills and goals jointly carried out by trainee and mentor that resulted in a work plan to address gaps in skills and contribute to the attainment of career objectives. Formal courses, self-study, and field missions, combined with mentor and mission team feedback, contribute to developing a broad range of skills necessary to become a senior consultant. The Guidelines on How to Prepare a Personal Training Plan for Future TB Consultants for TBCTA, developed by the Task Force on Training, was deemed to be extremely useful. While agencies managing apprenticeship programs did not all explicitly use this tool, some followed a process that closely approximated it. WHO and PAHO displayed the biggest weaknesses in mentorship and training plan development and implementation. If this apprenticeship program is to grow, clarifying the role of the mentor and the process of developing and monitoring the training plan will need to be standardized and formalized. The experts who were interviewed differed on their assessment of whether consultants from developing countries with program experience have better potential to become effective advisors than those from developed countries or those with a non-TB background. Most TB experts who were interviewed suggested that it might be easier to teach TB to an experienced international health consultant than to teach an experienced TB program person consulting skills. Less important to the success of apprenticeship programs is whether the program was headquarters or home-country based. Choosing trainees with the motivation and broad range of skills necessary to be a consultant, combined with a supportive mentor and well-designed training program, seemed to be more important than location of the apprentice during the training period. The two-week course to train international TB consultants in Sondalo, Italy, is well designed to simulate the actual experience of conducting a national program evaluation. The combination of technical presentations, role plays, group work, and report writing provides opportunities to develop the range of skills needed to become a consultant. Participants who were interviewed were extremely positive about the course. Suggestions to strengthen its impact include offering the most promising participants the opportunity to take part in a field mission, and creation of a database and a process to market qualified participants to potential clients. To guide USAID investments to train consultants, we recommend that a structured assessment of the current and future need for TB consulting support be carried out. A clear sense of the need for both general and specialized consultants will help determine the most appropriate selection criteria and training approaches that ensure that the need is met. This exercise is a “market analysis” that identifies demand (country need), current supply of general and specialized consultants, and the current and future gaps that need to be filled to increase supply to meet country needs. To do this effectively it will be important to clarify the “model(s)” of technical assistance that are most appropriate and cost-effective given global human resource constraints. Models that combine senior and more junior consultants and more effectively integrate the vi contributions of specialists in areas such as health financing, human resources, and social mobilization need to be considered. Long-term apprentice programs are intense, expensive, and not always responsive to emerging needs. Short-term courses are not enough to produce accomplished consultants. To increase the pool of consultants, the TB world will have to increase recruitment among less experienced staff, non-TB professionals, specialists, and experts with minimal consulting experience. Capacity to provide qualified technical assistance will have to be enhanced among a wider range of agencies than the current small group of acknowledged leading agencies in the TB field. To increase the number of organizations and individuals capable of providing quality technical assistance, a more targeted and lower average cost strategy will be needed. And since the ultimate goal is to build in-country capacity, there would be expected to be decreasing demand for external technical assistance over time. Four categories of trainees were identified, each requiring a different mix of training approaches. For example, a specialist who is an experienced consultant needs less focus on developing consulting skills and more on acquiring knowledge about TB. The specialist may benefit from participating in a relatively low cost web-based course, a short experience in a TB partner organization, several structured field experiences, and a formal course on TB program management. In contrast, an experienced TB program person needs more intensive training to develop consulting skills and might spend more time in a partner organization and focus less on the technical details of TB. This report recommends that the Task Force on Training or another entity examine the training needs of specific categories of potential consultants and design a targeted package of training methods to address the skill and knowledge gaps of each group. There are limits to the capacity of existing partners to support and eventually hire the number of consultants needed to support national TB programs. Another approach to consider is to fund TBCTA partners to train international consultants employed by other organizations that provide technical assistance in international public health to become TB experts. A model of training to consider is to have a range of Nongovernmental Organizations (NGOs), USAID Cooperating Agencies (CAs), and other organizations select a staff person to be trained to become a TB expert. TBCTA partners might agree to provide mentorship, supervised field visits, and some time based in the home organization, and the originating organization would commit to continuing to employ the trainee after the program was over. This approach would expand global capacity to support TB programs and would expand the number of firms with this capacity. vii I. Introduction The primary purpose of this evaluation is to inform future USAID investments to develop international TB consulting capacity based on assessments of a range of currently supported initiatives. In addition to the junior consultant training program supported by USAID and implemented by partners of the Tuberculosis Coalition for Technical Assistance (TBCTA), this evaluation looked at the apprentice training program run by the Pan American Health Organization (PAHO) and several short courses supported by USAID that specifically target training international TB consultants. The secondary purpose of this evaluation is to suggest measures to strengthen the current TBCTA junior consultant training program. Needs for technical assistance to TB control programs have escalated rapidly in recent years. Part of the change is driven by large increases in funding available for TB control from sources such as the Global Fund To Fight AIDS, Tuberculosis and Malaria (GFATM), USAID, bilateral and multilateral donors, and private foundations. Also impacting global assistance requirements is the HIV/AIDS epidemic and the need to develop coordinated approaches to TB and HIV/AIDS. Other trends that impact on demand for technical assistance include increasing resistance to first-line drugs and health system reform movements to expand Directly Observed Treatment Short-course (DOTS) within broader health systems. The challenge is that the capacity to provide technical assistance to countries has not grown rapidly enough to meet global needs. In response to these trends, in 2003 USAID began supporting a range of programs intended to increase the supply of qualified international consultants with expertise in TB control. Both intensive apprenticeships and short courses are being funded. While new consultants have been trained and the skills of existing consultants have been enhanced, the demand for technical assistance continues to far outstrip the supply. .Motivated to know what is working and to identify approaches that might be more effective, USAID asked PATH, through the Global Task Order, to review the performance, impact, and lessons learned of the Training Program for Tuberculosis Consultants under the TBCTA project and PAHO grant, both funded by USAID. The skills required to be a senior consultant are complex and take time to develop. In addition to knowing the technical details of TB, an effective consultant must be good at asking questions, listening, prioritizing, and communicating effectively, both orally and in writing, with people from different cultures and with different styles. As managing TB programs becomes more complex, there is a growing need for assistance from specialists in a wide range of areas. Most people interviewed believe that there is a need for both generalists who can take the “helicopter view” of a national program, informing policy and coordinating the inputs of specialists with a holistic vision much the way a general practitioner coordinates the interventions recommended by a range of medical specialists. Both generalists and specialists are needed, and effective approaches to training both groups require attention. It is also important to emphasize that skilled international TB consultants must frequently update skills because health systems change and the complexities of environments and the disease 1 evolve. This implies that developing the skills to be considered “senior” is a career-long undertaking. An apprenticeship program should upgrade skills and provide the framework to enable continued learning. It is not meant to be all that is needed to remain skilled over a long career. We looked at two broad categories of training programs. One group we refer to as “apprenticeship programs” where individuals are financed from 11 to 18 months in a program that combines formal courses, field experiences, self-study, and mentorship with the goal of developing the broad range of skills needed to become a senior consultant. We looked at programs financed through TBCTA and managed by TBCTA partners— The International Union Against Tuberculosis and Lung Disease (IUATLD), KNCV Fund (KNCV), and the World Health Organization (WHO) — and we looked at a program financed by a USAID grant to PAHO. In addition, we examined a short course, also funded through TBCTA, to train international TB consultants, that is run in Sondalo, Italy. We also attempted to learn about an additional short course run by the American Thoracic Society (ATS), also funded through TBCTA, for North American medical practitioners interested in learning about international consulting. Interviews with TBCTA partners identified an additional course, not included in this scope of work which was funded through TBCTA to train North Americans, which was run through the Web. We include some discussion of this course to present an alternative model to consider. This evaluation took a 360-degree approach to assess what is working and to identify areas for improvement. We read available documentation and interviewed trainees, trainers, mentors, supervisors, and clients. The broad questions we attempted to answer were grouped in the following categories: 1. How were people attracted to and selected for each program? 2. What were the expectations and objectives of trainees? 3. Were these expectations and objectives met? 4. What are trainees doing after training is completed? 5. What suggestions do people have for improvement? For short courses: 6. Do the curricula and training methods incorporate effective approaches to train adults? For apprenticeship programs: 7. How was the work program developed? 8. Did the implementation of the work program follow the plan? 9. How did mentors and supervisors function? While structured interview instruments were developed for each category of participant, we also encouraged the conversations to move in different directions to maintain the flexibility to identify unanticipated issues and suggestions. What follows is a description of the scope of work and the methodologies used in the exploration process. An overview of the consultant training programs that are the focus of this evaluation, which includes funding for each program and rough measures of cost-effectiveness, follows. Section V presents detailed discussions of each program, integrates feedback from interviews, 2 and draws on knowledge of effective strategies to develop human resources to suggest areas that are strong and areas for improvement. Section VI presents a range of recommendations to both strengthen the existing programs and to guide future USAID investments. II. Description of Scope of Work Recognition of a growing need for skilled consultants to assist national TB programs to strengthen their programs caused USAID to support programs to train consultants. This growing need is generated by the increasing complexity of managing an effective TB control program in the context of the challenges of coinfection with HIV/AIDS and expanding resistance to first-line drugs. New sources and levels of financing for TB control from the Global Fund To Fight AIDS, Tuberculosis and Malaria (GFATM), World Bank, USAID, and other donors also contributes to increasing the demand for qualified technical assistance. The purpose of this evaluation is to evaluate the impact of programs to train TB consultants funded by USAID through TBCTA and a grant to PAHO and to make recommendations for consultant training in the future. The following questions, included from the Scope of Work (SOW) included as Attachment I, were the focus of this assessment: Overarching Questions ¾ Are the consultants who have been trained actually working as consultants? If so, what portion of their time is dedicated to international consultancy? Are they helping to fill the technical assistance gap, and if so, how? ¾ Has the capacity of the individual organizations been strengthened by the program? ¾ What are the overall lessons learned and best practices from the training programs? ¾ Are the end objectives of combined training activities clear? Is there a strategic approach? ¾ What are the overall recommendations for adapting these programs to meet the consultancy demands in USAID priority TB countries and other high-burden countries? If possible, please try to factor in the cost-effectiveness of each activity. Technical ¾ Has the program contributed to the availability of higher quality technical assistance? ¾ Have the consultants met the objectives of their personal training plans? If not, what barriers or constraints prevented them from meeting the objectives? ¾ To what degree has the consultant training tool kit been useful for planning, monitoring, and evaluating the training of each consultant? For the mentor? For the trainee? ¾ Is there a need for training programs for specific technical areas, e.g., TB/HIV? ¾ How were the trainees selected? ¾ Have the learning objectives been subjugated to the day-to-day work needs of the organization? If yes, suggest recommendations for alleviating this issue. Dr. Christy Hanson, Director of the Global Health Task Order awarded to PATH, managed this evaluation. The consulting team included Dr. Rena Eichler health economist and independent consultant, and Donna Bjerregaard, human resources specialist with Initiatives, Inc. Information 3 for this report came from document review and interviews with trainees, trainers, clients, and USAID staff. What follows is a detailed description of the methodology, results, and recommendations. III. Methodology Information for this assessment came from a combination of document reviews and phone and in-person interviews. Before beginning the evaluation, the scope of work was shared with TBCTA board members for comments, and a session was held at the April board meeting to solicit additional input. Documents were provided by USAID and the TBCTA Project Management Unit (PMU). Additional documents were provided by people who were interviewed, when identified as valuable during conversations. Key information that was integrated into this report came from the assessment of the junior consultant training programs done at The Union meeting in Paris in October 2004 and evaluations of two sessions of the short course to train international TB consultants run in Sondalo, Italy. A list of documents that were consulted is included as Attachment II. Structured interview instruments were developed to use as guides during conversations. The intention was to be sure to cover a range of issues but to also allow conversations to focus on additional issues if relevant. Some trainees requested a copy of the questions in advance because of concern about communicating by phone in English, and two participants submitted a written reply by email. Questions covered a range of topics, such as how participants were selected, personal goals, development of a work plan, role of mentor and supervisor, assessment of the mix of learning opportunities, and recommendations for the future. Copies of the interview instruments are included as Attachment III. People were contacted by email and offered three suggested times and dates to participate in a one-hour conversation. If the suggested times were not possible, people were asked to suggest alternative times. Attempts were made to interview all participants in the junior consultant training program and all PAHO fellows, but due to lack of availability and, in some cases, lack of current contact information, 10 out of 17 participants were interviewed. Four out of eight PAHO fellows were interviewed, and six out of nine TBCTA junior consultants were interviewed. Seven mentors and supervisors of long-term training programs were also interviewed. Nine participants in the short-term Sondalo training course were interviewed, chosen to represent diverse countries and regions, along with three trainers and the course organizers. Anne Fanning, the developer of a web-based course to train TB consultants, was interviewed to get a perspective on another model of training. Though attempts were made to contact a range of clients who worked with junior consultants in the field, only one client, the National Tuberculosis Programme (NTP) manager of Egypt, was available. We believe that the reason other clients were not available is that the scheduling of interviews coincided with deadlines for countries to submit their Round 5 Global Fund proposals. Repeated attempts to set up phone and in-person interviews with ATS did not result in a meeting. Despite repeated requests, ATS did not share information about course participants, course curricula, or evaluations. For this reason, information about the ATS course given in Boston comes exclusively from other sources. Three representatives of USAID who oversee both TBCTA and PAHO programs were also interviewed. 4 IV. Overview of Consultant Training Programs and Funding This section provides a descriptive overview of funding and the training programs USAID has supported to train TB consultants that were included in this evaluation. Detailed assessments of each program are provided in Section V. Apprenticeship Programs (TBCTA junior consultant training and PAHO fellowship program) Since 2003, USAID funded international TB consultant training programs that were implemented by four organizations. Three programs were implemented by TBCTA partners— KNCV, The Union, and WHO. The fourth program was implemented by PAHO through a USAID grant. There are important similarities and differences between these programs that provide opportunities to compare approaches and to identify those that are most effective. The programs can be considered “apprenticeship” programs because in addition to formal training courses and reading (self-study), they include a considerable amount of learning by doing. WHO and PAHO programs last 11 months, the KNCV program lasts one year, and The Union program is 18 months. This section presents some differences to help frame the discussion that follows. Analysis of the implications of these differences is included in the sections describing each program and in the section presenting recommendations. Do all the apprenticeship programs look for people with a similar background? Because selecting appropriate candidates is critical to the success, this evaluation compared the criteria used to select apprentices across programs. Partners used a range of selection criteria to choose candidates. PAHO’s selection criteria specify people with at least three years’ experience working in TB control at the national or local level, as well as training in public health or the equivalent. PAHO looked for people with skills in their own country who would be valuable for the LAC region. The Union also chose people with TB control experience, but also emphasized the interpersonal skills needed to be an effective consultant. Both PAHO and The Union selected candidates from developing countries. In contrast, KNCV believes that it may be better to select someone with international consulting experience in public health, but with limited TB control knowledge. KNCV believes that it is easier to teach an experienced consultant TB than a knowledgeable TB program person to be a consultant. KNCV selected people from both developed and developing countries as did WHO. WHO selected one specialist with no TB experience and with human resource development (JRD) expertise and one generalist with experience providing assistance to TB programs. There was no expectation that participants in the PAHO program would be available to work full time as international consultants after the training program ended. Most PAHO participants were on leave from their positions and returned after the training program ended with the expectation that they would be available for periodic short-term consultancies with PAHO. In contrast, TBCTA apprentices were being trained with the expectation that they would become full-time international consultants. One TBCTA supervisor was clear that he did not view international consulting as a part time job. KNCV promised participants the opportunity to be hired after training was completed if performance was good. While WHO could not make the same 5 commitment, there was an expectation that efforts would be made to continue with either short￾term or possibly fixed term contracts. The Union was not able to make commitments but is trying to find funding to hire their trainees. Common Selection Criteria Used Developing country national? Experience in TB control? International consulting experience? Available to work as an international TB consultant full time when program ends Full- time employment promised if performance good? The Union yes yes no yes no KNCV no no preferred yes yes PAHO yes yes no no no WHO￾general consultant no yes yes no no WHO￾HRD specialist no no yes no no What process was followed to identify and choose trainees? TBCTA partners and PAHO publicized the training programs on various web sites, advertised in newspapers, and contacted networks of people in the TB field. WHO and PAHO also sent requests to regional offices to ask for recommendations. In the first round, KNCV received and reviewed 40 applications. A selection committee interviewed a short list of candidates by phone. Two candidates were selected in the first group; they have since graduated. One had a strong TB background with a history of consulting in Europe and Central Asia; the other was an experienced consultant in community-based care programs, with little TB background. Recently, KNCV recruited an HRD specialist. They had hoped to be able to identify a candidate with expertise in HRD and international consulting experience in public health. As the recruitment process did not identify any candidate with all of these qualifications, KNCV decided to train a very junior person with an HRD background but no knowledge of international consulting, public health, or TB. The Union selected the current three trainees from a pool of 58 applicants. The finalists met both professional qualifications and The Union’s interest in engaging people with diverse geographic and language backgrounds. There was a desire to have one trainee from English-speaking Africa, another from French-speaking Africa, and an additional person from some other region. Chosen trainees included a person from India, Zimbabwe, and Cameroon. Candidates also had to demonstrate a degree of independence and motivation to substantiate their unsupervised home country placement. WHO sought two candidates: one TB generalist and one junior person with international and international organization experience and with some experience in training to become an HRD expert. One hundred and thirty applications were evaluated for the HRD position. The chosen 6 HRD specialist was Australian. WHO decided to give the generalist position to WHO-Euro, who recommended and chose the trainee. She is originally from Russia and had been a WHO-Euro TB Officer before the training program. Potential PAHO Fellows submit a detailed application listing their professional education and experience, language skills (English), computer literacy, fellowships and awards, association membership, publications, and references, as well as short paragraphs on their areas of interest. The prerequisite for English and computer competency was dropped after it proved to be a deterrent to otherwise able candidates. It was replaced by a pre-Fellow language and computer course undertaken in the candidate’s home country. Currently two people are financed through the USAID grant to PAHO to build English language and computer skills in preparation to be a Fellow in 2006. Qualified applications are reviewed by a committee composed of representatives of the PAHO Communicable Diseases Program and USAID Bureau for Latin America and the Caribbean, and Bureau for Global Health, Office of Health, Infectious Diseases and Nutrition, who judge the candidate’s potential skills for addressing weaknesses in the Americas Region, and seek a balance of countries and gender. The program also strives to bring other professions into the field; a nurse and microbiologist have been among the trainees. Where was the apprentice based during the training period? The Union felt is was important to recruit people from developing countries and to continue to have them based in their country to provide ongoing support to the country NTP. Mentoring takes place through email, phone conversations, joint missions, and quarterly consultant meetings in Paris. The supervisors of this model emphasize that it is important to select highly self-motivated people because they can not be as closely monitored as those in a headquarters based model. The Union was able to lengthen the training program to 18 months and train three rather than two people, given their budget, because the costs of basing people in their home country are lower. KNCV, PAHO, and WHO move their apprentices to headquarters, where they manage mentoring, training, and coordination of field visits. It is important to note that while PAHO apprentices spend most of the 11 months in Washington, DC, they recently added a 2-month internship program based in Brazil, where they work closely with the NTP. KNCV has recently hired a consultant from a developing country who is based in a different developing country during the training period. Location of Apprentice During Training office? The Union yes No no no ( no yes yes no no Based in home country? Based in HQ or regional Based in HQ with extended field experience? KNCV yes no and yes Kutwa) PAHO WHO yes 7 How are training plans developed? When apprentices in TBCTA partner programs began their program, they were intended to follow a process to develop a training plan that was developed by the TBCTA-supported Task Force on Training (TFT). The tools used to guide the training plan development process are Guidelines on How to Prepare a Personal Training Plan (PTP) for Future TB Consultants for TBCTA and a format for documenting the plan. The guidelines are based on the premise that effective training can neither be a one-time event nor standardized, as trainees come from diverse backgrounds with different strengths, learning styles, and consulting objectives. The direction, tools, and resources, including an annotated list of appropriate courses and study tools, are included in the guidelines tool. The guidelines recommend a collaboratively developed plan by the trainee and his/her appointed mentor and consists of: a) reviewing candidate competencies against a list of required competencies for TB consultation; b) identifying training or experiential gaps; and; c) identifying appropriate courses and peer meetings, study guides, and field missions to develop the required skills. The final plan acts as a contract between the consultant candidate, who agrees to what he/she will do to develop the competencies needed to become a consultant, and the supervisor, who outlines the support the consultant candidate will receive during the training period. It is meant to be a “living” action plan and a monitoring tool for both the candidate and the supervisor to review progress and revise the plan. Once completed, the intention is for the apprentice to independently implement the plan and meet regularly with his/her supervisor to review progress. The supervisor is expected to set up field missions that incrementally demand more independence and participation from the participant. This is arranged between the supervisor and “preceptor” or mission supervisor before the mission. Supervisory meetings are also meant to assess progress and any need for revisions to the action plan. TBCTA partner organizations varied in the approach they used to develop training plans. How much has been spent on consultant apprenticeship programs? For the three years beginning in 2003 and continuing into the current year, USAID has spent over $1.5 million to train 17 international TB consultants in long-term training programs through TBCTA partners and PAHO. The average cost of training per consultant is $88,278. Of the 17 trainees, 8 are still in training (3 The Union, 3 PAHO, 2 KNCV). There is some variation in cost to train consultants among programs, but differences are partly due to specific details of each program. For example, KNCV recently recruited a relatively junior HRD specialist and an Africa-based junior consultant into the program, making the average cost to train the four KNCV consultants appear lower. What are trainees who have completed the program now doing? Of the nine who have completed their programs, three have been hired by their training organization as international TB consultants, two are employed by international organizations and working as TB control consultants, two returned to former jobs and are interested in short￾term consultancies, and two are no longer working in TB control. If working full-time as an 8 international TB control consultant is an indicator of long-term success, then the average cost of producing a full-time TB consultant under these combined programs has been $158,900. USAID Funding for Consultant Apprentice Programs (2003-2005) Budget (Project years 3,4,5) Number of people trained or in training Average training cost per person Number completed training Number who completed training now working as full￾time TB consultants PAHO consultant training 1 $649,598 8 $81,200 5 2 The Union consultant training $279,111 3 $93,037 0 WHO consultant training $216,508 2 $108,254 2 1 KNCV consultant training TOTAL $355,500 $1,500,717 4 17 $88,875 2 2 $88,278 9 5 Average cost to produce a full time international consultant $158,9002 Short Course to Train International TB Consultants Through TBCTA, USAID has funded a two-week course four times, run in Italy, to train international TB consultants. The intention is to select participants with a strong knowledge of TB control so that the focus of the course can be on developing consulting skills. Participants are recommended by WHO regional offices, other international organizations, and NGOs and come from around the world. The course is designed to simulate the experience of conducting a national program review. It uses a combination of technical presentations, role-playing, feedback, group work, and report writing to mirror what happens in an actual field mission. While the goal of the course is to prepare people to be consultants for WHO, the skills learned can be applied in a range of consulting activities for other clients. 1 Not included is $28,870 to train two trainees who are receiving language and computer skills training in their home country in 2005 in preparation to be a Fellow in 2006. This portion of the budget also funds participation in courses and international conferences. 2 = ($88,278* 9)/5 9 Of the 55 participants who attended the USAID financed courses, 15 (27%) came from the 22 highest burden countries (HBCs). Participants come from across the globe as can be seen from the following table. Funding from USAID paid for travel costs and per diems for all participants and instructors, honoraria for the course organizers, and materials. Average cost per participant to attend the course (not including travel from home country to course) was $3,469. Travel costs were estimated to be an average of $2,000 per participant, making the average cost per participant roughly $5,469.3 Total spending on four sessions of the Sondalo course by USAID was $301,816. Country of Origin of Participants in the International TB Consultant Training Course Number of participants Africa 9 3 14 8 1 0 9 1 2 0 4 0 3 2 6 1 7 0 55 15 Transportation costs for 55 TOTAL Number from high burden countries Asia Central Asia Eastern Europe Middle East North America Pacific Region South America Western Europe USAID Funding for the Short Course to Train International Tuberculosis Control Consultants in Sondalo, Italy Budget (course given 4 times) Average cost per participant Site costs (per diems, materials, honoraria for instructors, etc.) $190,816 $3,469 Management costs for KNCV $11,000 $200 participants $110,000 $2,000 $311,816 $5,669 3 Source: TBCTA PMU communication. It is not clear where these additional costs are reflected in the budget table provided to this consultant team in the SOW. 10 Have participants worked as international TB consultants since completion of the short course? A request was sent in April 2005 to the 55 people who participated in the four sessions of the TB consultant training course supported by USAID to update contact information and to monitor whether people have worked as international consultants after the course. Of the 55 requests sent, 32 people responded (62%). Of the 32 people who responded, 27 worked as international consultants (79%) after they attended the course. It is important to note that 21 out of the 27 who reported having done an international consultancy since they took the course did so for the first time. It is not clear from available information whether these new international consultants provided technical assistance multiple times or only once. It is also not possible to fully attribute this outcome to the course, but it is an indication that the right people were selected for training. These data provide some structure to make rough estimations of the cost-effectiveness of this short course under two scenarios. The low case scenario assumes that out of 55 course participants only the 27 that replied worked as international consultants after the course, making the average cost of strengthening the skills of an international consultant $11,178. The high case scenario would estimate that the same percentage of nonrespondents worked as international consultants as the respondents. Under this high case, 44 out of the 55 participants would have worked as international consultants after the course, making the average cost to strengthen the skills of an international consultant $6,859. The true number is probably somewhere in between these two extremes. In addition, it is important to emphasize that it is not possible to attribute work as an international consultant after participation in the course exclusively to the course. Estimated Average Cost to Produce a TB Consultant in the Sondalo Short Course Under Two Scenarios Effectiveness scenarios Average cost Low case scenario (27/55) $11,178 High case scenario (44/55) $6,859 Courses to Train North Americans to be International TB Consultants Through TBCTA, USAID funded two additional programs to train TB consultants from North America. One of the programs, managed by ATS, was included as part of this evaluation. As previously said, it was not possible to schedule a meeting or phone interview with representatives of ATS, so limited information included in this report comes from other sources. In addition, we were not sent key materials from ATS that would be necessary to evaluate the selection process of participants, the course content, or the impact of the course. Through interviews of others involved in training, we learned of a web-based course to train North American consultants that was also funded through TBCTA. We followed up with the course organizer to understand more about the course and to try to identify methodologies that might have been used in the course that could benefit international consultants. 11 ATS course to Train North American Consultants. To our knowledge, ATS ran one short course in Boston aimed at teaching North American medical practitioners interested in doing occasional international consulting how to provide technical assistance to TB control programs in developing countries. Other interviewees commented that the course was weak and that it was not possible to train people with no international consulting experience to be international TB consultants through this vehicle. Interviewees suggested that “hobbyists” can’t provide quality assistance because becoming an international consultant requires a substantial professional investment. Without additional information it would be hard to evaluate the impact of the course, but interviewees suggested that the majority of the participants in the course do not work as international consultants after the course is completed. Total funding to ATS for training through TBCTA was $78,000 between 2003 and 2005. Web-based course to train international consultants. Through TBCTA, USAID also financed a web-based course to train North Americans to be international consultants. The course ran for one year and contained 11 modules that covered technical and programmatic components of a TB program review. It was managed by Dr. Anne Fanning of the University of Alberta. The course used a blended learning model that included readings, interactive web-based sessions, and on-line facilitation by leading TB control experts. Course content addressed the essential components of TB control program rationale and application, with key references and relevant examples. Participants were recruited from the association of TB controllers the North American Region of the International Union Against Tuberculosis and Lung Disease, and by word of mouth. Of the 30 who expressed initial interest, 20 participants began the course and 11 completed the course, including all the assignments. The first time the course was implemented was intended to be a test of the approach, with the hope that feedback would be used to refine materials and the approach so that a future group of international consultants could participate. Modules were facilitated by leading international experts and covered the following areas: epidemiology, interventions, country and situation analysis, planning and implementation of programs, program evaluation, management, Global Fund, HIV issues, lab, TB in prisons, training and supervision, multi-drug resistant (MDR) TB, and advocacy and social mobilization. The majority of course participants rated their satisfaction with the quality and usefulness of each module from “somewhat satisfied” to “very satisfied.” Additional challenges were that the workload was more intense than participants had expected and some found it hard to access the course on-line when traveling. In addition, participants provided the qualitative feedback that they would also benefit from field experience to directly support what they were learning and provided the input that opportunities for field experience would be more valuable than a week of face-to-face classes. From participation in this course, three participants were able to participate in field missions to Sudan and Myanmar, and one has been employed as a consultant directly related to the opportunity afforded by the course. The overall budget provided by USAID for development and implementation of the course was $100,000. No TBCTA funding was approved for subsequent years.4 4 Source: Communication with Dr. Anne Fanning and draft paper provided by Dr. Fanning describing the Web￾based course. 12 V. Assessment of Apprenticeship Programs and Short Courses to Train International Consultants This section presents the results of a detailed assessment of the apprenticeship programs supported by USAID and implemented by the TBCTA partners, followed by a discussion of the PAHO apprenticeship program and the short course to train international TB consultants run in Sondalo, Italy. TBCTA Apprenticeship Programs What is the program goal and is it attainable? The goal of the junior TB consultant training program is to increase the availability of competent senior TB consultants who can respond to the global need for technical assistance in planning, evaluation, and implementation of TB control programs. It is important to emphasize that the qualities of a senior consultant are not clearly defined and are difficult to assess and that it takes more than a time-limited training program to fully develop a senior consultant. Years of field experience working in a variety of contexts, combined with formal and informal training and continued learning, are required to produce a fully skilled senior consultant. In the past, technical support was provided by the TBCTA partners in the form of monitoring missions, program reviews, and other technical assistance. In recent years, however, the growing demands of increased funding, demands of multiple donors, and the push to meet the Millennium Development Goals for TB have confirmed that there is a shortage of human resources with the skills needed to provide technical assistance to help countries meet the challenges of a growing epidemic. Through its long-term training approach and combination of personalized training methodologies, skill practice in structured field assignments, and supportive supervision, the apprenticeship program provides a vehicle for developing senior TB consultants. To what degree has the consultant training tool kit been useful for planning, monitoring, developing, and evaluating the training of each consultant? How have different organizations interpreted and supported mentorship and what approaches have been effective? Has the combination of self-study, formal courses, and field missions contributed to the development of apprentices? The three TBCTA implementing partners have implemented the Guidelines on How to Prepare a Personal Training Plan (PTP) for Future TB Consultants for TBCTA in different degrees. KNCV followed the Guidelines closely and adapted and revised them to suit individual needs. WHO followed the process in the initial design phase but did not appear to refer to the developed PTP to monitor and revise individual trainees’ programs. The Union did not follow the explicit process but appeared to incorporate the “spirit” of the Guidelines in the way they approached assessment of trainee needs and development of the training program. The effectiveness of the training plan development process is dependent on adhering to the principles that underlie the Guideline, combined with good communication between the players. This requires a committed, well-oriented mentor who has thorough knowledge of the competency requirements; capacity to assess the candidate’s current skills, background, and 13 experience; and up-to-date knowledge of the resources available to address any identified gaps. Additionally, the organization must support the dedication of staff time for the mentor’s activities and provide frequent updates on appropriate resources, including courses and field missions. Finally, the PMU should act as a repository of information and regularly share information about applicable program resources and be informed of the need for new courses to address emerging gaps. KNCV. Both of the now graduated KNCV consultants used some form of the Guidelines to develop a personal training plan to assess knowledge and skills and to design the personalized program, which were reviewed with their mentors. Courses were identified from the listing provided in the Task Force on Training (TFT) Guidelines as well as through the Internet. Field missions were well structured, and discussions with mentors strengthened the experience. The recently recruited junior HRD consultant also developed a training plan with the PTP guidelines but found it difficult to adapt to her training needs. It is too soon in the process of her program to fully evaluate whether her training program will be adapted to meet her needs. KNCV attempts to link mentors to trainees on the basis of availability, regional compatibility, and seniority. However, the mentorship role is not clearly defined. Given that mentors play a key role in building capacity and satisfaction among apprentices, clarifying the role of mentors and the skills required to be effective would strengthen the process at KNCV and with all evaluated apprenticeship programs. In the early stages, when the numbers of apprentices being trained is small, quality mentoring appears to have been provided in spite of this lack of formal specification of the responsibilities of mentorship. Placement at headquarters is an advantage as it gives the candidates wider access to a range of technical KNCV staff, a significant factor, as the demands of travel can affect the maintenance of a relationship with a dedicated mentor. This loosely structured approach will be less effective if the program grows and the range of specialists widens. A case in point is the HRD specialist chosen in November 2004. After trying in vain to recruit an experienced international public health consultant with HRD expertise, KNCV decided it was worthwhile to offer a professional trained HRD specialist with experience in other industries and train her to develop missing capabilities. Without TB experience or international consulting experience in public health, her training will focus on building her HRD skills and general consulting skills to encompass TB issues and needs. She will be blazing new territory in KNCV’s junior TB training program as the organization and the trainee collaboratively attempt to assess her TB consulting learning needs, define her role, find appropriate field missions, and integrate her into the organization. Trainee perspective: The two interviewed graduates were pleased with the plan development process and the balance of training activities. The consultant with TB experience was able to define his skill needs and collaboratively participate in developing his learning plan and identifying a range of courses and study materials from networking and online research. He was extremely satisfied with the support received from his mentor. The trainee less conversant with the TB world was also satisfied with the training received and appreciated the possibilities that exist within KNCV to build on her previous experience to develop specialty areas such as social mobilization and community DOTS. After recognition that the initial mentor assigned to this 14 trainee was not providing adequate support, KNCV assigned a new mentor who was appreciated and valued. While mentorship could have been more structured and the quality of the standard courses could have been better, she was very satisfied with her training experience. Both candidates stated that field assignments played a significant role in building relevant skills and fostering independence. In addition, they endorsed the headquarters placement for its link to learning opportunities but also recommended combining it with an extended field experience, especially for those candidates with a minimal TB background. One graduate suggested that an extended placement (internship) in an organization such as UNAIDS would have contributed to developing desired expertise in TB/HIV. The new candidate will face similar challenges. She has already spent time adapting the guidelines to meet her distinct needs. She would like clarity on expectations for her future role as an HRD specialist and how the organization can respond to her needs for TB exposure. Although she wants to participate in general TB consulting missions to round out her knowledge, a mission has not yet been organized. . The Union. The Union did not explicitly follow the Guidelines, but the principle of reviewing current trainee competencies against the ideal list was followed. Mentors were assigned and assisted their candidates to identify courses and reading material and undertake relevant missions, as well as provide input to their operations research. Candidates receive strong support from mentors through email, missions, and in meetings held quarterly in Paris. Significantly, The Union has a dedicated supervisor/coordinator to oversee the program in addition to pairing each trainee with a seasoned mentor. The program supervisor emphasized that she dedicates a lot of time to support trainees, coordinate missions, and communicate expectations to field mission preceptors as well as trainees. The benefit of this structure can be seen in the positive response by the trainees; the home country-based placement probably demands this support. There is recognition that both general senior consultants who can provide broad technical assistance and specialists with focused skills in areas such as human resources, financial management, and health systems are needed. The perception of the characteristics of consultants required to support national programs has evolved to needing “niche” consultants as well as overall generalists. This may affect selection and training in the future. If the program is to grow, there needs to be more focus on supporting the supervisor/coordinator’s role in this distance learning approach and may possibly require a dedicated person to manage the program. Additionally, the distance requires a more structured approach to using the guidelines and developing an action plan that provides the tools for implementation review and evaluation. Trainee perspective: The interviewed Union participants gave the program extremely high marks. The support and guidance received developed skills and confidence. Action plans were created without the benefit of the personal training guidelines but relied on the same principles and competency review. The result was an accurate assessment of consultant needs and identification of appropriate courses, literature, and field mission assignments. The field assignments were planned to demand progressively more independence as experience was gained. The home-based candidates defended their home country placement but recognized it relied on self-motivated trainees. One candidate pointed out that access to operations research was more 15 difficult if one was not also attached to an organization in the home country. The readiness of mentors to provide feedback and the ability to participate in quarterly Union meetings in Paris provided enough opportunity for feedback, information sharing, and planning. WHO. WHO supplied a third variation. Based on information from the one available WHO apprentice, the PTP assessment process was not formally followed, but the approach used followed the TBCTA recommended format for assessing existing competencies, identifying gaps, and developing a PTP. The process and plan was shared with peers and the mentor and feedback was incorporated to revise the plan. It was challenging, however, to fully implement the plan as designed because of the realities of WHO work life emphasizes supporting WHO activities and the regional autonomous structure made it difficult to orchestrate field opportunities in other regions. Implementation of the plan resulted in a combination of “learning by doing” and structured training. Training comprises courses, peer-group meetings, field missions, and self-study as well as on-the-job learning. The consultants evaluating the programs were only able to interview one of the two WHO apprentices because one is no longer working in TB. The WHO-Euro generalist was recommended for the program by her supervisor and she continues to be supervised by him in her present role. This may have posed challenges for ensuring the most productive mentor/trainee relationship. She conducted her own assessment of her learning needs and shared it with her supervisor. The supervisor was then responsible for advising on the components of the training, providing feedback and guidance when requested, and assisting in identifying relevant field missions. This was done by helping the trainee select priority areas and linking her with relevant WHO specialists, who provided study material, missions, and experience in organizing workshops. The supervisor was surprised to learn that he was the assigned mentor in this training program, as no expectations for the mentorship role were communicated to him. Trainee perspective: The interviewed trainee expressed appreciation for the opportunity to participate in the training program and its contribution to her professional growth. While she was clear that she found her supervisor/mentor to be extremely supportive, it does not appear that the support provided throughout the training period was consistent with the goals of the apprenticeship program. The mentor gave feedback and support when approached and gave input into the development of the training plan when asked. He did not identify field missions in other regions that corresponded to the trainee’s skill development needs and learning objectives, perhaps because of a lack of clarity within WHO about how this was expected to happen, as regional autonomy within WHO complicates activities involving more than one region. As a result, the trainee’s desire to have more exposure to ethnic and cultural diversity by gaining experience outside of the region, particularly in an African country with a high prevalence of HIV/AIDS, was not addressed. Lack of clarity about the objectives of the apprenticeship program and the roles of mentors in the process was also reflected in information received through other sources about the HRD junior TB consultant specialist, who left the organization after completing her second contract. She reported dissatisfaction over the lack of a personalized training approach, weak mentoring and being subjugated to the operational needs of the organization. The objectives of the apprenticeship program did not appear to be clearly communicated to those responsible to support apprentices nor to the apprentices. 16 Has the program contributed to the availability of higher quality technical assistance? The objective was to develop the skills of junior TB consultants so they can become senior consultants and to increase the pool of senior international consultants available to strengthen national TB control and prevention programs. Within KNCV and The Union the satisfaction level of both staff and trainees was high; there was clarity about the purpose, goals, and the strategies, which were agreed upon collaboratively. Trainees felt the individualized approach and dedicated mentoring helped them develop personally and professionally. To date the two graduates of KNCV training have been hired by the organization and two of The Union trainees are hopeful, though nothing is yet guaranteed, that they will have similar results at The Union. In addition, the programs have exposed the candidates through their field missions to a host of other organizations that can potentially employ them. It is important to note that no strategies to help the apprentices find employment have been implemented, should The Union be unable to secure funding to hire them. TBCTA Junior Consultant Training Participants Organization Participant Country of Country Prior TB Specialization Current origin based experience status KNCV Kamp Netherlands Netherlands No Community KNCV employee Dara Iran/Belgium Netherlands Yes KNCV employee Van Sanvoort Netherlands Netherlands No HRD In training TBD TB/HIV Recruiting The Union Wilson India India Yes In training Dhliwayo Zimbabwe Zimbabwe Yes In training Abena Cameroon Cameroon Yes In training WHO Yurasova Russia WHO Copenhagen Yes WHO employee Sheppard Australia WHO Geneva No HRD trainee program focused on No longer working in TB development of HRD control competencies. In contrast, the structure of WHO that renders regional offices relatively autonomous makes it hard to organize experiences for trainees in other regions. In addition, the lack of clarity about program objectives and the role of mentorship in WHO hampered the ability of the program to adequately focus on apprentice development. Although WHO professes to use a “learning by doing” approach to build skills, it does not follow that methodology’s principles. Used by training programs in health as well as other areas, the approach is based on doing, reflecting, conceptualizing, and experimenting to be sure the lesson is learned. In this context the emphasis would be on setting up experiences to meet the individual’s objectives, reviewing them with an experienced mentor, and applying the learning in new situations. To truly implement a “learning￾by-doing” approach, WHO would have to prioritize the PTP guidelines, commit itself to reaching the defined competencies and learning objectives, assist in providing relevant learning 17 experiences, strengthen mentoring, and structure activities to build on previous learning. This may require a structured process to identify institutional constraints to attaining these goals and strategies to either overcome constraints or alter the model of apprentice training employed. If the junior TB consultant training program is to expand, the three organizations will have to institutionalize the process of assessment, training design, evaluation, and marketing of new consultants; similarly they will have to train senior staff in mentoring skills and objectives. Standardization will contribute to more focused training and support, skill development, evaluation, and sustainability. PAHO Apprenticeship Program What is the goal of the program and is it attainable? The USAID/PAHO Training Program in Tuberculosis aims to build consulting capacity in the Latin American and Caribbean (LAC) region to deliver technical assistance for TB prevention and control programs. The goal is appropriate and achievable with the right combination of applicants and structured learning experience from the organization. An expected by-product of the program is bringing the region into the global exchange of strategies and achievements, which has mutual benefits. Given the exposure that the trainees will gain through publications and attending international workshops, this is also attainable. Carefully selected participants are expected to have gained the following competencies in the area of TB after 11 months of training: • Operational knowledge. Demonstrated knowledge of DOTS implementation and health improvement and prevention priorities. • Tuberculosis knowledge. Demonstrated knowledge of TB program prevention and control processes, policies, partners, and programs at the community, country, regional, and global levels. • Research. Understanding of and ability to apply scientific research principles and techniques. • Surveillance and data collection. Capacity to design TB surveillance data collection tools/materials; ability to collect, analyze and interpret data; and knowledge of statistics. • TB programs o Planning o Development o Management o Implementation o Evaluation • Advocacy. • Community engagement. • Team skills. • Personal leadership. • Networking skills and partnership development. • Public-health surveillance and research. 18 These objectives are manageable within the timeframe with a comprehensive assessment of trainee knowledge and skill and learning objectives and the right balance of appropriate professional activities, field missions, training and mentorship, and self-determination. It should be recognized that the program structure and training approach followed by PAHO does not prioritize the development of individualized training plans. Does the training methodology contribute to high-quality technical assistance? The program was initiated in 2003 in response to the lack of TB expertise within the Latin American Region. The then PAHO TB expert took on promising young candidates with TB experience as apprentices. Through a variety of field missions and structured supervision, the apprentices were helped to understand and respond to the technical assistance needs of LAC countries. This “learning-by-doing” approach was successful with the original Fellows who benefited from the leadership, mentoring, and commitment of the program’s initiator. Satisfaction and growth through this process were based on adherence to the principles of this approach, including (a) clear knowledge of competencies required, (b) identification of trainee’s learning needs, (c) emphasis on building knowledge (d) structured missions to practice skills, and (e) evaluation of progress. Unfortunately, in the short life of the program, PAHO Fellows have had three different mentors; subsequent directors transformed “learning by doing” from its emphasis on strong mentorship and applicable field missions into an immersion experience in the work life of the organization. Without relying on personalizing training plans or reflective supervision, emphasis is placed on report writing and presentation skills in English, project management, proposal writing, progress reports to donors, and exposure to other health issues. The trainees are stationed at PAHO headquarters in Washington, DC, where they can learn about PAHO operations, partake in meetings and conferences, and find appropriate missions in which to participate. This then develops their skill to take on short-term PAHO assignments. Fellows are expected to participate in at least three field missions and pick one technical area to study during their tenure. Candidates spend two months under the direction of the program’s initiator, Rodolfo Rodriquez, who is now the WHO TB Advisor in Brazil, systematically learning about the health system and participating in field missions. There are no current guidelines for developing a collaborative personal training plan; current PAHO leadership, in fact, suggested that a structure for defining learning objectives, a general list of skills required, and an individualized program to meet those needs identified would be helpful. As in the TBCTA program, the appropriateness of the capacity development plan is dependent on the commitment of the organization to ensure a learning environment and the capacity of the supervisor to provide support for skill development. PAHO needs to define its goal, training approach, and the staff attention required to give the program the structure it is obviously lacking. Participant perspective: Four Fellows were interviewed. There was a divergence of experience and satisfaction, particularly reflecting the different phases of the program. Although all 19 candidates benefited from participation in missions, the cohorts in the second and third programs remarked they were treated more like PAHO team members than trainees being groomed for that status. Participants in the first cohort expressed satisfaction with their personal training plans and weekly supportive meetings. Subsequent groups were less positive, citing the absence of a learning needs assessment, responsive structured personal workplans, and individualized attention to personal development goals. Fellows felt they were participating in too many field missions where they were immersed in assessment missions as fully contributing consultants without time for feedback from mentors; the most significant comment was an expressed uncertainty over whether PAHO was really developing their skills. Has the program contributed to the availability of higher quality technical assistance? The two main objectives of the program are to build and improve consulting skills and to contribute to the needs of the region. In the three years this program has been operating, eight Fellows have been engaged. Of these, three are still in the program, two returned to their previous positions and are available for short-term consultancies, one returned to the position he held prior to the training program and does not participate in international consultancies, and two have been hired full time as international consultants by other organizations. The program has since been hampered by its lack of commitment to a structure or even a coordinator, creating confusion for both mentor and learner. Although skills have improved through participation in missions, the training approach has not provided the structure to systematically address trainee knowledge and attitude. Called “learning-by-doing” by the organization, it is more apt to describe the process as an immersion course into PAHO operations and activities in the field of TB, a by-product of which is learning the demands and practice of consulting. It is also not clear that “graduates” are promoted for use of PAHO or other organizations in the field. To continue supporting this program, it would be prudent to measure the impact of the approach on trainee growth and development and ultimately employment. 20 PAHO Fellow Status Participant Country Gender Full-time availability for Specialty Current Status consultation Zavala Peru M Yes Employed by CIDA to run a project in Ecuador. Luna Chile F No Lab Returned to Chile. Available for short-term consultancies re lab and MDR TB surveillance. Tenorio Colombia M No Returned to Colombia. Performs on-line consultancies for reviewing or preparing technical documents and short-term consultancies to support regional TB meetings. Hernandez Mexico M No Epidemiol- Returned to position in Mexico. ogist Reyes El M Yes Union consultant in Brazil Salvador Echegaray Bolivia M No In training Jacquet Haiti Yes In training Montero Brazil F No Nursing In training Short Course Through TBCTA, USAID funded several short courses aimed at training international TB consultants. As mentioned earlier, this evaluation team was not able to get information directly from ATS about the course they ran for North American medical practitioners. Some information was acquired about a web-based course run by the University of Alberta that was discussed briefly in Section IV. The focus of this section is on the WHO Training Course for TB Consultants run at the WHO Collaborating Centre for Tuberculosis and the S. Maugeri Foundation in Tradate, Italy. This course will be referred to as the “Sondalo course.” Are the workshop objectives attainable? The goal of this two-week training workshop is clear, concise, and achievable. The objectives, however, are ambitious for the timeframe and the scope of the training. The workshop is part of a broader strategy to “further develop the necessary skills to plan, implement and evaluate a TB control programme, based on the WHO recommended TB control strategy, DOTS.” The intense course immerses participants in the role of WHO technical advisors and exposes them to the skills necessary to plan, implement, and evaluate TB control programs. Through learner and facilitator assessments of participation growth in exercises, meetings and discussions, and the required mission report; overall skill development can be assessed. 21 Achieving the objectives depends on selecting the right participants and measuring individual growth. As the course is designed for TB technical people who need additional programmatic skills in planning, implementing, and evaluating TB control programs, appropriate participants are those almost ready for international consulting. Goals state that at the end of the training, participants will leave with the competency to act as “consultants at the national and intermediate levels of TB control programmes to support countries in different activities and to be recruited as WHO staff, and even more ambitiously, participants will be able to: • Plan, prepare, and manage a WHO mission. • Manage the key elements of TB control, from drug purchasing to implementation of pilot projects. • Manage and organize a laboratory network. • Train, negotiate, and communicate. • Prepare a report in the headquarters format. Achieving the first objective depends on participants’ previous professional experience, technical knowledge, and ability to adapt to the diplomatic and technical requirements of consultation. In fact, facilitator evaluation confirmed only 37% of the 27 participants evaluated in the first two times the course was run as ready for independent consultation. It would be difficult to demonstrate achievement on actual skills identified in the second objective without having the “graduates” participate in a mission, manage a TB control program, organize a laboratory, and conduct training. It would be helpful to review the second objective and restate the outcomes in terms of what can be measured during the course of the training; at best the participants will have practiced the skills. Does the training methodology contribute to skill development and competency? The course is cleverly designed to simulate the actual experience of participating in a country mission, from the preparation phase to analysis to delivering the mission report and recommendations. Housed appropriately in the Morelli Hospital, a former tuberculosis sanatorium in Sondalo, Italy, workshop participants become temporary WHO advisors for 14 days. The roughly 15 participants per workshop come from different continents and backgrounds, which contributes to the learning and provides for dynamic exchanges and new insights into the challenges of managing TB control programs. Most participants have extensive experience in country or regional NTP programs and broad knowledge in the field of tuberculosis. Some are more specialized, having worked in drug management or laboratories. The training is designed to acquaint participants with the role and tasks of general TB consultants and have them practice the skills they learn through role plays, group work, field visits, and exercises during the day and report writing by night. The tight schedule is intended to mirror actual time-pressure and conditions found in field assignments. Highly experienced consultants from WHO, the Collaborating Center (Sondalo) and University faculty, lead the program. The course revolves around a case study of the fictitious, but realistic, Afro-Asian country of Fictitia at present and five years hence. The time span gives the participants a chance to practice 22 skills related to the development of a TB control program and advise on implementation challenges in a more advanced stage of the program. Presented with their Terms of Reference (TOR), the “consultants” are charged with organizing the team and logistical arrangements as they prepare the mission. To acquaint them with WHO rules and guidelines, the consultants adhere to administrative procedures and documentation requests as they would on a mission. The participants are provided a participant’s manual and oriented to the expectations of their mission and course deliverables. The units focus on developing the technical assessment and interpersonal skills necessary for consultation. The teaching material is usually presented in a PowerPoint format, complemented by a group or role-play exercise to give the “consultant” an opportunity to apply what they are learning while building competency in negotiating, problem solving, team building, leadership and communication. Using the case study, participants systematically review NTP plans, surveillance data, laboratory network organization and quality assurance, procurement cycle, human resource organization, community organization and advocacy, budget adequacy, partner coordination, and operations research. The facilitators and trainees engage in the role-plays; those not directly engaged in the exercise serve as observers and are actively involved in the debriefing sessions, an important part of the learning process. Participants are expected to attend meetings, debrief the Ministry, and ultimately prepare a written report with their recommendations. The units are logically arranged to enable synthesis of the material and determination of appropriate recommendations to strengthen the program in the final report and debriefing. Additional units are devoted to current themes, such as TB-HIV collaboration, TB in the private sector, and to The Global Fund to Fight AIDS Tuberculosis and Malaria (GFATM) proposal writing. Methodology. The course never deviates from its adherence to adult learning principles. The facilitators are helpful and respectful, providing supportive feedback to each participant. The teaching methodology is participatory and interactive, allowing participants to learn from each other and through applying the knowledge in exercises. The material is engaging and challenging, as it recognizes and builds on participant prior knowledge. Learners are forced to use and analyze the material to reach the objective of their course participation: developing the competency to be an international consultant. Does the course contribute to expanding and strengthening the global TB consulting pool? Aside from the ambitious objectives, the main weakness in the design of the course is the lack of post-training support. USAID should consider funding post-training support to strengthen the impact of its investments. Effective training structures include teaching, practicing, and doing. Although simulated exercises mimic the reality of consultation in the workshop, there is no follow-up to allow consultants to participate in evaluation missions structured to address their technical and presentation gaps and for which they would receive constructive feedback. It is important to note that the facilitators have created a database of participants to be shared with WHO offices to encourage referrals of the newly trained consultants. The participants are 23 aware of this resource and are told that they may be called for assignments based on workshop recommendations. However, funding limitations have not allowed promotion of this resource list for a wider network of agencies. Does trainee selection contribute to reaching workshop objectives? Given the demands of the course and the expectations for trainee performance, the selection criteria are both appropriate and necessary for realizing the expected results. WHO headquarters and regional offices are asked to identify candidates who have the knowledge and skills to plan, manage, and/or implement a TB control program and professional credentials that demonstrate this competency, as well as the potential to be consultants. Staff or consultants of international organizations are considered along with participants from the TBCTA junior TB consultant training program. Participation also results from individual, nonaligned applicants who usually learn of the training through personal recommendations. Most candidates are NTP managers or high-level NTP staff or have leadership positions in NGOs. Their technical background sets the stage for developing competency in interpersonal communication, leadership, and oral and written presentation skills in the workshop, essential for consultants. The combination of the TB control program expertise, which the trainees possess, and the understanding and practice they gain in the workshop setting should assist them to meet the workshop goal. However in some cases when experts in a narrower field, such as logistics, drug management, or laboratory, were selected, the intense curriculum could not compensate for the lack of a broader background in TB. Although the training benefited these candidates, their presence forced adaptations in the content and flow of the program and compromises in the final outcomes. How are participants evaluated? Two main criteria are used to evaluate participant achievements: their ability to (a) examine and correctly judge the positivity or negativity of three direct smear slides and (b) produce a well￾structured, clearly written, comprehensive report which fully uses the information provided in the case study and exercises and makes relevant and realistic recommendations. In addition, workshop facilitators assess trainee participation in group work, role plays, and discussions, looking for appropriate skill in communication, conflict management, cultural competency, team building, problem solving, and leadership for fundamental competencies and knowledge and skill in assessing health systems and health policies, epidemiology of TB and HIV, TB control programs, TB program needs and resources, surveillance data, diagnostic and treatment services, drug management, human resource and financial management, research, IEC, program management and implementation, and evaluation and advocacy to elicit support for DOTS expansion. Based on facilitator assessments, feedback provided to the participant includes an analysis of their readiness to be an independent consultant and, if appropriate, any follow-up training needs. Do participants feel their consulting knowledge and skills have increased? 24 After each of the four times the course was run, feedback from participants was solicited through a written evaluation; the input was used to strengthen the course format and material. The feedback from the first course led, for example, to the introduction of Fictitia part two and the removal of the evening sessions. The results showed participants either strongly agreed or agreed that most units were effective and relevant. Given the general high evaluation scores, it is important to note that the units of Planning, Report Preparation, Budgeting, and Supervision received less than a 70% effectiveness rating5 in both courses, perhaps indicating a need for review. During the present assessment, interviews were conducted with nine course members and three facilitators. The participants concurred with their earlier evaluation; their purpose in joining the course was to become a consultant and the format, exercises, and international mix of participants contributed to reaching that goal. They appreciated the experience in writing a report and receiving feedback and the chance to see the steps of a mission unfold from preparation to finalization. They acknowledged that although the course was intended to produce consultants capable of joining WHO, the skills learned were useful for preparing to deliver technical assistance in general. They were also highly impressed with the competence of and support received from the facilitators. They did make suggestions, which support the findings of this assessment: • Consider basing the course on an actual country evaluation, structuring the time to allow review and feedback for each step. • Limit the course to experienced TB experts; do not include those with more limited or specialized experience as this slows down the course. • Reduce length of course. There was an underlying feeling that the course was too long. The second week was not as sharp as it suffered from participant and facilitator fatigue. • Use the database to promote the graduates as potential consultants in their country, region, and in general where their skills are needed. • Consider adding an opportunity to participate in a field mission. • Conduct a course in French to reach potential consultants not fluent in English. Does the workshop contribute to expanding the availability of high-quality technical assistance? The workshop fits into a broader program to increase the number of competent TB consultants able to provide technical assistance to countries with a high burden of tuberculosis. By a facilitator evaluation based on assessments of two of the four times the course was run, 6 of 26 participants are ready to participate in missions, while another 4, due to lack of language skills or familiarity with countries outside of their home, are competent to provide country or regional technical assistance; 3 could provide technical assistance in specific service areas. In total, 38% had met the objective of being able to provide broad technical assistance, while 50% needed 5 Bergstrom, Karin, Final Report on Sondalo submitted February 2004. 25 additional experience. It is also important to emphasize that some of the TBCTA junior consultants participated in this course and in other courses offered in Sondalo. Sondalo Facilitator Assessment Suitable for international consultation Suitable for country/regional activities Suitability for specialized consulting (drug, lab) Requires more experience (on the job, mission, broader country) Total evaluated 6 4 3 13 26 This judgement is based on facilitator observations and evaluation of the written reports. However, funding limitations prevent the provision of post-training support to strengthen the skills of the remaining 50%, as well as the promotion of the participant database to enable recruitment of the other consultants by relevant organizations. In the final analysis, the time and funding for preparing the “consultants” is reported by participants and facilitators to have increased individual skills and should lead to higher quality technical assistance within their present employment. It is more difficult to assert, without pre￾workshop knowledge of participant international consultation experience, whether the course is directly responsible for increasing the pool of international consultants. What is known is the data reported in Section IV that shows that of 32 respondents to a recent query sent to the 55 people who have participated in this course, 27 report having done recent international consulting. VI. Recommendations This section provides recommendations to USAID in order of suggested priority to guide future investments in expanding global TB consulting capacity. After this section, suggestions to strengthen current programs are offered. Recommendation 1: Develop a model to estimate the global need for both general and specialized TB consultants. We recommend that a structured assessment of the need for TB consulting support that estimates the need for generalists and specialists be carried out. A clear sense of the need for consultants will help determine the most appropriate combination of selection criteria and training approaches that ensure that the need is met and will contribute to developing a training plan to guide future USAID investments. A structured assessment that estimates the need for generalists and specialists would contribute to determining the most appropriate combination of selection criteria and training approaches. This will ensure the need for TB consultants is met and will contribute to developing a training plan to guide future USAID investments. This exercise is a “market analysis” that identifies demand (country need), current supply of general and specialized consultants, and the current gaps that need to be filled to increase supply to meet country needs. It is important to both reflect the situation today as well as to forecast needs of the future, given epidemiological, funding, and health systems trends. 26 To do this effectively it will be important to clarify the “model(s)” of technical assistance that are most appropriate and cost-effective given global human resource constraints. For example, it will take more time to train people if the model assumes that all members of an evaluation team are senior consultants. Another model to consider includes having a team leader who is senior combined with a team of mid-level generalists and targeted specialists. It will also be important to consider changes that are happening in NTPs that may impact on the future need for external technical assistance. For example, Indonesia and India have effectively scaled up implementation of DOTS by training a cadre of health officers to provide technical support to TB control programs at the regional level. Further adoption of this model would also impact on projections of the type and quantity of external TB consultants required in the future. The TB community is encouraged to look at team models used in other areas of public health and to consider approaches that can be implemented rapidly. Models that more effectively integrate the contributions of specialists in areas such as health financing, human resources, and social mobilization are also needed. This exercise can build on past estimates developed by The Union that assumes that there is a need for one international general consultant for every 25 million people in high burden countries, looks at the current supply of consultants, and identifies the gap. The model should look at both the need for general consultants and specialists who can contribute in key areas and should consider alternative approaches to comprising technical assistance teams that combine less experienced with seasoned people as a way to amplify capacity and provide support to countries. The assessment model should answer the following questions: 1. What are the areas of specialized technical assistance needed by TB control programs and how many specialists will be needed now and in the future? a. What are the areas of specialization needed? i. TB/HIV ii. DOTS Plus iii. Drug management iv. Laboratory strengthening v. Community mobilization vi. Advocacy vii. Private-public mix (PPM) viii. Human Resource Development (HRD) ix. Preparing donor proposals (Global Fund for AIDS, TB, and Malaria, other) x. Practical Approach to Lung Health (PAL) xi. Integration of TB into the primary health care (PHC) system xii. Incentives and enablers xiii. Operations research xiv. Health financing xv. Infection control xvi. Other? b. How many should be senior? c. How many should be mid-level? 27 d. How many are needed with specific regional and language expertise? e. How many senior specialists in key areas are needed? f. How many specialists in key areas already know TB? How many specialists would be interested to learn TB? A model can be developed to simulate the need for consultants with different profiles under a range of assumptions. Clients should be surveyed to identify their assessment of needs and their preferences. The program priorities of dominant funders should also be assessed. A working group of senior international TB experts can review the survey approach, assumptions, and the projections under each assumption to reach consensus on the global need. Answers to these questions can identify priorities for training and can ensure that a comprehensive plan can be developed to address global needs. Part of the framework underlying the analysis in this report is that there is a “market” for technical assistance in tuberculosis control with the countries as the “consumers” that determine the demand and the consultants and organizations that provide technical assistance as the “producers” or suppliers of assistance. One aspect of this market that is recently changing is that countries are making their own decisions to hire consultants to provide technical assistance and are managing the payment directly with sources of funding for TB control such as the Global Fund. While there are many reasons that this market does not work the way a model of pure competition would predict, some elements of a market response would be predicted that are either not happening, not happening rapidly enough, or happening in ways that are potentially dangerous. Some of the questions worth asking include: 1. Why does the expertise appear to be concentrated among a small group of large international organizations rather than moving toward an expanding group of organizations with TB expertise? 2. As funding for TB control increases and more NGOs and consulting firms move into the TB field, what are the incentives for the small group of agencies that traditionally provide technical assistance to strengthen the capacity of others to provide technically sound advice? 3. Given this growing gap between demand and supply of qualified technical assistance, are there other models to staff teams and provide technical support that are more likely to meet global needs? Are we seeing changes to this approach? While the scope of this evaluation is to assess the impact of approaches to training TB consultants, the consultant team feels that to truly understand the challenge of enhancing the availability of technical assistance it is also important to consider why people and organizations are not moving more rapidly to satisfy the growing demand from countries. We present the following possible reasons to help frame the discussion that follows: 1. It takes a long time to become a skilled TB consultant, especially because the DOTS approach is rigorous, and an inadequate number of individuals are interested in making this investment. 28 2. There is some understandable resistance to change in the organizations that traditionally have provided technical support. 3. While the traditional providers of technical assistance recognize the need to train and integrate consultants with areas of expertise needed by TB programs, they have not yet figured out how to provide the appropriate structure and support to do this effectively. 4. Current funding from USAID and others is not providing the incentives to support the expansion of technical capacity in a broad range of potential partners. The current traditional providers of technical assistance do not have mechanisms or financing to strengthen the capacity of NGOs and other providers of technical assistance. Even if answers to these questions indicate that there is resistance to change, USAID is making a wise decision to try to speed up the process of training and employing skilled consultants because the need is great and people are dying needlessly. The pressing challenge is to determine what works best and is most effective at quickly generating a large pool of qualified people available to provide needed assistance. Recommendation 2: Develop a training plan that is most cost-effective at creating the global capacity estimated by the model. Estimations of global need for both specialists and generalists and the composition of technical assistance teams could contribute to the development of a strategy to determine how to best invest USAID resources. An expert should examine the range of activities and intensity of support that will be needed to develop each target group and contribute to the development of a training plan. Long-term apprentice programs are intense, expensive, and not always responsive to emerging needs. Short-term courses are not enough to produce accomplished consultants. To increase the pool of consultants, the TB world will have to increase recruitment among less experienced staff, non-TB professionals, specialists, and experts with minimal consulting experience. Capacity to provide qualified technical assistance will have to be enhanced among a wider range of agencies than the current small group of acknowledged leading agencies in the TB field. To increase the number of firms and individuals capable of providing high-quality technical assistance, a more targeted and lower average cost strategy will be needed. For example, feedback from various sources stated that assisting experienced consultants to become TB experts is simpler than the reverse. Creating teams of senior and junior consultants will facilitate a faster, more robust response to target countries. Training specialists to apply their expertise to TB control will fill an urgent need felt by TB program managers. Each category of trainee requires a different mix of training methods to build required skills. A combination of mentorship, web-based courses, distance learning, short focused workshops, and experiential missions to practice learning could be an effective strategy to train various categories of consultants. The following table outlines some scenarios to build technical capacity. 29 Alternative Scenarios to Build Global TB Consulting Capacity Target Group Objective Rationale Suggested training methodology Experienced TB consultant Focus on 2 months in partner organization with Non- TB developing TB focused technical skill development consultant technical skills Complementary web-based courses to cover technical areas 3 Structured field missions Experienced consultant who is also a specialist (non-TB) Specialist with TB slant Focus on developing understanding of TB program needs 2 months in partner organization with focused technical skill development and integration of specialization into ongoing TB work Complementary web-based courses to cover technical areas Technical courses on TB program management (e.g., managing DOTS at district level or Sondalo course) 2 structured field missions TB mid-level Junior Focus on 6 Months in Partner Organization with program consultant developing the focused technical and consulting skill knowledge with technical and development no consulting interpersonal experience skills needed for Sondalo Course consulting 2-month field assignment to learn country needs 3 structured field missions Junior consultant Senior consultant Focus on taking leadership role in consultant 3 months in partner organization Sondalo course missions, report writing, donor coordination 3 structured field missions; lead role in 2 missions These four categories and the methodologies suggested to train each group are presented as illustrative. USAID could consider prioritizing funding to train international consultants based on estimation from the model. It is likely that training specialists with international consulting experience mentioned in the top row will be a high priority. Approaches to train more junior people with knowledge of TB but less consulting experience will also likely be a priority, especially if models to provide technical assistance will evolve to combine senior and junior people. It is important to recognize that it will be challenging to organize and schedule 30 participation in courses and field missions and that a degree of flexibility as well as considerable planning and coordination will be needed to implement this approach. Future USAID investments in training should be informed by the assessment of current apprenticeship and short-course programs. As currently implemented by TBCTA and PAHO, there are three distinct junior consultant training program models, one in which candidates are based in the headquarters, the second in which trainees are assigned to headquarters but also spend two months in the field, and the third in which trainees remain in their home country and meet quarterly with peers and mentors in headquarters. The programs vary in their orientation to international consulting and objectives, selection criteria, training approach, and program structure. With the limited number of graduated trainees it is not possible to conclusively say that one model is superior to others. For example, the home country model appears to be successful if the people chosen are highly self-motivated and if strong mentorship is provided. Headquarters￾based models have other advantages, including the opportunity for frequent interactions with a team of professionals working in TB control. All approaches have benefits as well as challenges that must be considered during implementation. The following table attempts to capture the factors that facilitate and impede the capacity building experience as reported by the participants and their mentors in the current programs, with a goal of informing future USAID investments. 31 Training Approach Analysis of Existing Apprentice Training Models Centralized vs. Field-based model Org. Length (months) TB Number of participants Strengths Weaknesses Trainee satisfaction 12 3 ƒ ƒ ƒ ƒ Headquarter ƒ corrected ƒ Headquarter￾11 2 ƒ ƒ ƒ ƒ areas of WHO ƒ ƒ ƒ ƒ Headquarters + field 11 ƒ ƒ ƒ ƒ ƒ ƒ ƒ ƒ ƒ Low 18 3 ƒ ƒ ƒ ƒ ƒ ƒ background? KNCV Not required Personalized training plan Well-structured field assignments Frequent peer exchange and exposure to other experts placement Inconsistent mentorship that was More difficult to provide adequate support for non-TB trainees High to medium based WHO Not required Field missions Opportunity for peer exchange Exposure to WHO orientation Exposure to other expertise, including HIV/AIDS, health systems, etc. Weak mentorship Unclear objectives Poorly defined training strategy WHO institutional constraints that inhibit ability to work across regions Medium PAHO Required PAHO orientation Field missions Brazilian extended field assignment Weak mentoring No personal training plan Lack of peer exchange Lack of structured training Lack of program structure No requirement to be a full-time TB consultant post training Field-based Union Required Strong mentorship Personalized training plan Peer exchange Home country placement No employment guarantee Need organizational affiliation in￾country High Assessment of the current apprenticeship training programs does lead to some recommendations that should be considered by USAID for future investments in consultant training. Recommendation 2.1: Future apprenticeship program should select candidates who are committed to international consulting as a career. Providing skilled technical assistance is not a part-time job or a periodic opportunity for “hobbyists.” As TB program managers become more sophisticated and running a national program becomes more complex, seasoned and committed consultants will be needed to provide both broad and specialized support. Current long-term apprenticeship programs are too expensive to train people who will only be available periodically. If a model is implemented that is less costly, it may be possible, for example, to train specialists who provide assistance in their area of specialty to TB and other programs. Recommendation 2.2: Specify the role of mentors, choose mentors with strong mentorship abilities, and orient them to the goals and objectives of the program and their role, and allot adequate time to provide support. The most significant facilitating factor in current apprenticeship programs was the presence of a dedicated, consistent mentor, who made time to help trainees assimilate their learning; strove to collaborate to find experiences to meet their needs; and provided guidance on self-study material, reports, and field assignments. Organizations that chose mentors who were relatively unresponsive or were not clear of their role contributed to frustration on both sides. In the future, the role of mentor should be specified, the qualities of a strong mentor should be defined, and appropriate people should be assigned the role. Adequate time should be allotted to the mentorship role to ensure that trainees receive strong support. Recommendation 2.3: Use the personal training plan, or a modified approach, to match the objectives of the global training plan to identify competencies, individual learning needs, and a customized training strategy to reach them. The PTP approach developed by the Task Force on Training was judged to be extremely effective when implemented properly. The process of assessing candidate needs and identifying capacity-developing activities to address those needs in apprenticeship programs that followed resulted in a strong individualized training experience. We recommend building on this approach to develop a modified PTP that is appropriate for the range of training objectives determined by the training plan. Recommendation 2.4: Encourage organizations to develop a plan to best integrate trainees. The plan could include actions to orient general staff about the structure and purpose of the program and could clarify the roles of mentors, supervisors, and peers. Clarity about the objectives of the program is essential to guide the structure and activities. Lessons from currently supported apprenticeship programs indicate that organizations that took time to identify the objectives and parameters of the training program also selected the most ardent and focused mentors, who participated in the assessment process and took care to find and orient appropriate missions to the needs of the trainee. In the clearest adherence to this process, The Union appointed an overall supervisor for the program who saw that trainees were in a structured, personalized training setting. Recommendation 2.5: Institute monthly discussion groups on emerging topics, technical updates, and new findings. The presence of peer exchanges was considered a benefit by trainees for their development and satisfaction in current junior consultant programs. Future USAID￾supported programs should encourage the development of peer exchange networks, through intranets and periodic meetings, across organizations to enable learning and strengthen links among international TB consultants. The following recommendations draw from assessment of the short courses to train international TB consultants currently funded by USAID. Recommendation 2.6: USAID could strengthen skills by supporting a field experience as a member of an assessment mission following the formal Sondalo course. Effective training always involves a practice and application phase, spaced closely together. This would solidify the theoretical learning and be helpful to those who need more practice before establishing themselves as international consultants. Recommendation 2.7: USAID could build on existing or develop a new web-based course for full time TB apprentices who are specialists or who come from other areas of public health. Participation in a web-based course similar to the course implemented by the University of Alberta may help to fill in some of the technical gaps expressed by specialists and consultants without TB experience. It may form a core part of the global training plan if an effective and flexible course can be developed and facilitated. Recommendation 2.8: USAID could increase the availability of French-speaking consultants by funding the Sondalo course for French-speaking participants. If the assessment of the global need for TB consultants indicates a strong need for French-speaking consultants that cannot be met with the current pool of consultants, USAID should consider funding a version of the Sondalo course in French. While all participants who were interviewed who attended the Sondalo course enthusiastically appreciated the opportunity to share experiences with people from TB programs across the globe, there was a recognized need to train people from French-speaking Africa who are not comfortable enough in English to fully participate. Recommendation 2.9: Do not support short courses for North American medical practitioners who are interested in infrequent international consultancies. Senior international TB experts who were interviewed agreed that international consulting is not for “hobbyists” but for full-time committed people or people planning to make a career change that will involve transition into significant international work. Short courses aimed at training North Americans engaged in domestic work to become international consultants on an infrequent basis were not deemed to be effective. 34 Recommendation 3: Leading TB organizations that provide technical assistance could mentor staff from other agencies to increase the number of agencies with the capacity to provide high-quality assistance to TB control programs. There are limits to the capacity of existing TBCTA partners to support and eventually hire the number of consultants needed to support national TB programs. Another approach to consider is to fund leading TB organizations that provide technical assistance to train and mentor international consultants employed by other organizations that provide technical assistance in international public health to become TB experts. A model of training to consider is to have a range of NGOs, USAID CAs, and other organizations select a staff person to be trained to become a TB expert. Mentoring organizations might agree to provide mentorship, supervised field visits, and some time based in the home organization, and the originating organization would commit to continuing to employ the trainee after the program was over. This approach would expand global capacity to support TB programs and would expand the number of organizations with this capacity. To effectively implement this recommendation, USAID will need to consider the incentives mentoring organizations will face to assume this training and mentoring role. Currently, TBCTA partners have the incentive to mentor people who may become future employees. USAID will have to implement additional conditions to ensure that mentoring organizations devote sufficient resources to train and support apprentices who will strengthen the capacity of other, potentially competing organizations. However, many of these organizations are already working on TB at the community level and could complement the larger TB organizations’ efforts. Recommendation 4: Strengthen the coordination, networking, and quality control role of the agencies chosen to implement USAID-funded training programs in the future. Recommendation 4.1: Strengthen the orientation and coordination role of the executing agency to respond to the needs of the long-term TB consultant training programs. As the program grows and there is more variation in personalized training plans to meet the needs of a diverse group of trainees, responsibility has to be taken for collecting and disseminating information about new courses and potential field missions, for assisting the trainees to communicate across organizations, and for developing and supporting new courses to cover emerging needs and gaps. The executing agency could also orient participating partners to the intent and structure of the PTP process and ensure that it is revised and improved upon by integrating feedback about its effectiveness as it is applied to a diverse range of trainees with different needs and from varying backgrounds. Recommendation 4.2: Strengthen the role of the executing agency to promote recruitment of consultants. To facilitate placement of TB experts into full-time positions or to promote matching of country program needs to trained or “accredited” consultants, the executing agency could develop a database, website, and strategy to market consultants to international and national organizations. USAID could ask the instructors of the Sondalo course to formalize the evaluation criteria to provide recommendations about the level of skills and areas of expertise of each graduate. To make this course effective those who “pass” could be made known to the international TB community. Newsletters, websites, and general emails to relevant technical assistance organizations would also help. 35 Recommendation 5: Strengthen the ability of USAID to monitor implementation progress and program success. Recommendation 5.1: To monitor the quality of training programs, USAID could require mentoring agencies to report on feedback from clients and leaders of field missions about the consulting skills of apprentices and trainees from short courses such as Sondalo. Current evaluation of apprentices includes no formal evaluations or feedback from clients and colleagues who participate in field missions. USAID could require reports on the effectiveness of apprentices that evaluate skills in technical and consulting areas. Feedback from clients and mission leaders would also help mentors adjust the training process to address skill gaps. If USAID supports field missions for promising participants in short-term courses such as those taught in Sondalo, feedback from clients and mission leaders would be extremely valuable to assess the quality of the course as well as the effectiveness of consultants. Recommendation 5.2: To track performance and adherence to a structured apprenticeship program, USAID could monitor indicators such as: adhering to agreed upon selection criteria, number of consultancies conducted, number of supervisory sessions, number of client assessments conducted, and number of graduates in full-time consulting roles. Recommendation 6: What follows are suggestions for USAID and TBCTA to consider to strengthen the current junior consultant training program that are feasible given the short time frame: Recommendation 6.1: Orient mentors to the goals and objectives of the program, and, their role in the process,s and allot adequate time to provide support. In the period remaining in the TBCTA cooperative agreement, the role of mentor could be formalized, the qualities desired in a mentor could be identified, and guidelines could be communicated to the partner organizations. Recommendation 6.2: Institute monthly discussion groups on emerging topics, technical updates, and new findings. The PMU could implement monthly discussion groups among junior consultants and recent “graduates” through an intranet or other web-based approach to facilitate learning and sharing among trainees. Recommendation 6.3: Develop a client and mission leader survey to assess performance of junior consultants and to assess future needs to contribute to thinking about future USAID investments in training and to guide the development of a global training plan. Current evaluation of apprentices includes no formal evaluations or feedback from clients and colleagues who participate in field missions. USAID could encourage TBCTA to develop a client and mission leader survey. This information should be shared with mentors and supervisors and should be used to periodically revise PTPs. In addition, feedback from clients is an opportunity to receive input on future consulting needs based on client perceptions. 36 Attachment I: Scope of Work Scope of Work Evaluation of the Training Program for International Consultants May 2–June 10, 2005 I. INTRODUCTION The U.S. Agency for International Development, Bureau of Global Health, Office of Health, Infectious Diseases and Nutrition (USAID/GH/HIDN) seeks the services of Program for Appropriate Technology in Health (PATH) under the Global Health Task Order to review the performance, impact, and lessons learned of the Training Program for Tuberculosis (TB) Consultants under the Tuberculosis Coalition for Technical Assistance (TBCTA) project and Pan American Health Organization (PAHO) grant, both funded by USAID. II. BACKGROUND Globally, TB is a problem of enormous dimensions. In 2000, it was estimated that there were about eight million new cases of TB worldwide with two million deaths. TB kills more people than any other single infectious agent except the human immunodeficiency virus (HIV). Ninety￾five percent of all TB cases and 98 percent of TB deaths occur in developing countries. However, TB also has an impact on industrialized countries with certain groups, especially those persons born in high TB incidence countries. TB has a major impact on economic development both in countries with a high incidence of the disease as well as globally. In developing countries, 3-4 months of family income or up to 20-30 percent of annual household income can be lost due to TB. The potential cost to a nation due to lost productivity from TB is estimated between 4 and 7 percent of GDP. Not only does TB exact a heavy toll in human suffering and economic losses currently, it is increasing every year. A major contributing factor to the incidence of TB is infection with HIV. Today about 15 million people have dual infections with Mycobacterium tuberculosis and HIV. In many African countries, where TB is often perceived as being synonymous with AIDS, more than one half of TB patients are HIV-positive. In some countries, the HIV epidemic has led to a tripling or even quadrupling in the incidence of TB during the past ten years. There are twenty-two countries consisting of 80 percent of the global TB burden. Strategy for Tuberculosis Control: Global TB control is possible through a well-established strategy known as DOTS (Directly Observed Treatment Short Course). The DOTS strategy recommended by the World Health Organization (WHO) and the International Union Against Tuberculosis and Lung Disease (Union) has now been adopted by 119 countries worldwide. It is one of the most cost-effective health interventions and is rigorously promoted around the world. The global targets of the DOTS strategy for the twenty-two high burden countries is to reach 85 percent treatment success and 70 percent case detection by 2005. 37 Implementation of DOTS requires a strong primary health care system that ensures sustained access for TB patients to diagnosis, treatment, and follow-up services. Important elements of such a system include a network of capable laboratories, a recording and reporting system, and a robust logistics system that ensures a secure supply of drugs. Political commitment to fund and implement effective national tuberculosis control programs is essential to promote global TB control and avoid the consequences of failure. In addition implementation of DOTS requires improved access to primary care services that are affordable, equitable, committed, and well organized. Education and training are essential elements to ensure the availability of human resources. Not only are high-quality human resources required at the country level but experienced consultants are necessary to assist developing and transitioning countries control TB and meet the global targets. With all the additional financial resources for DOTS expansion in the last few years from USAID, the World Bank, the Global Fund, and other bilateral donors, the demand for quality technical assistance has become an urgent issue. USAID is one of the lead international organizations to give attention to building TB consultant capacity to meet the need for technical assistance worldwide. Currently, USAID is supporting two consultant training programs in collaboration with TBCTA and PAHO. USAID also supports other activities in this area; however, they are not the focus of this evaluation. The Tuberculosis Coalition for Technical Assistance: The Tuberculosis Coalition for Technical Assistance (TBCTA) is a USAID-supported project, executed through a unique partnership of six organizations6 actively involved in global TB control. One of the main objectives of TBCTA is to expand the capacity for providing high quality technical assistance worldwide, particularly in developing and transitional countries. In response to the growing need for TB consultants, TBCTA developed a program to recruit and train junior TB or non-TB consultants to become senior TB consultants in June 2003. The program is based on the Guidelines on how to prepare a personal training plan for future TB consultants for TBCTA (http://www.tbcta.org/Publications/guidelines.php), developed by the TBCTA Task Force on Training (TFT). The main premise of the program is to provide individual training programs to match the specific needs of each consultant since their background and experience may vary. TBCTA partner organizations, KNCV Tuberculosis Foundation, the World Health Organization (WHO) and the International Union Against Tuberculosis and Lung Disease (The Union), have recruited and started training seven junior consultants to take part in the program (see Annex 1 for list of junior consultants). Each consultant is paired with a mentor who conducts an initial assessment and helps develop a personal training plan using the guidelines. In October 2004, a questionnaire was conducted and a meeting held as an informal assessment of the program. Current TBCTA partners: American Lung Association (ALA), American Thoracic Society (ATS), Centers for Disease Control and Prevention (CDC), International Union Against Tuberculosis and Lung Disease (The Union), Royal Netherlands Tuberculosis Association (KNCV), and the World Health Organization (WHO). 38 6 In addition to the junior consultants’ program, TBCTA has supported various training activities to build the capacity of TB consultants. Over the past three years, TBCTA has co-funded a course to increase the international TB consulting capacity in the United States. In addition, TBCTA has co-sponsored several training courses for TB consultants at the WHO Collaborating Centre in Tradate (Sondalo), Italy. There have been a few additional consultant training activities supported through the TBCTA project. The complete list can be found in Annex 2. Pan American Health Organization In 2003, USAID and PAHO began a fellowship program to strengthen the human resources capacity in the Latin American Region to deliver technical assistance for tuberculosis prevention and control to developing countries. It is an eleven-month program where trainees are located at the PAHO office in Washington, DC, as well as supported, supervised, and mentored by this office. To date, five trainees have completed this program and three are currently being trained in the program (see Annex 1 for list of trainees). Since 2004, there has been an alternative preparatory program supporting professionals at country level. During the first year (2005) the applicant will learn English language and computer skills in his/her country. She/he will participate in meetings and monitoring visits to country. During the second year (2006) the candidate will be incorporated in the training program in Washington, DC. It is expected that by the completion of the training period, the trainees will have acquired and strengthened their skills in key areas such as operational knowledge, tuberculosis knowledge, research, surveillance and data collection, TB program design, planning, monitoring and evaluation, advocacy, community engagement, team skills, personal leadership, and networking and partnership development skills. III. OBJECTIVES The evaluation will synthesize and analyze the data collected as well as make recommendations regarding consultant training for the future. While the evaluation is not intended to be exhaustive, it is expected to address the key questions outlined below. The consultant will look retrospectively at the two programs and answer the questions outlined in Section III and make recommendations for the future. The results will be used to provide feedback to USAID and partners as well as provide advice on the most effective way to build consultant capacity to meet the TB technical assistance needs in the future. IV. KEY QUESTIONS As a guide for this activity, key questions for the evaluation are outlined below. However, all aspects of the two consultant training programs could be evaluated in order to determine its impact and future direction. 39 Overarching Questions ¾ Are the consultants that have been trained actually working as consultants? If so, what portion of their time is dedicated to international consultancy? Are they helping to fill the technical assistance gap, and if so, how? ¾ Has the capacity of the individual organizations been strengthened by the program? ¾ What are the overall lessons learned and best practices from the training programs? ¾ Are the end objectives of combined training activities clear? Is there a strategic approach? ¾ What are the overall recommendations for adapting these programs to meet the consultancy demands in USAID priority TB countries and other high burden countries? If possible, please try to factor in the cost-effectiveness of each activity. Technical ¾ Have the consultants met the objectives of their personal training plans? If not, what barriers or constraints prevented them from meeting the objectives? ¾ To what degree has the consultant training tool kit been useful for planning, monitoring and evaluating the training of each consultant? For the mentor? For the trainee? ¾ Is there a need for training programs for specific technical areas, e.g., TB/HIV? ¾ Has the program contributed to the availability of higher quality technical assistance? ¾ How were the trainees selected? ¾ Have the learning objectives been subjugated to the day-to-day work needs of the organization? If yes, suggest recommendations for alleviating this issue. V. METHODOLOGY The evaluation will gather information through interviews with: 1) TBCTA and PAHO staff, trainers and supervisors of consultants (approximately 10 interviews); 2) junior consultants and other course participants trained by partners (approximately 20 interviews); and 3) recipients of technical assistance by trained consultants including NTPs, USAID missions, and other technical agencies (5-10 interviews). The selection of interviewees will be based on the availability of personnel and through consultation with USAID. A standard questionnaire for structured interviews will be developed. The evaluation will also review course objectives and curriculum, course materials, participant selection criteria, participant evaluations, and mission reports. Sources of information and methods used will be described in the final report. Document Review § USAID/Washington will provide the PATH consultant with historical program documents before the planning meeting. These documents will include the list of junior consultants and other course participants, project work plans, individual training plans, project assessments, course objectives and curriculum, training materials, mission reports and any other relevant materials. § The TBCTA and PAHO partners will be a good source of information for additional documents on specific activities. These documents will be made available upon request. § The PATH consultant will be responsible for collecting and reviewing any other relevant documents throughout the evaluation. 40 Briefing Session § A planning meeting will be held in Washington or by telephone before the evaluation begins. The meeting is important for the following reasons: -enabling USAID personnel, such as the CTOs, TB Team, and others to present the team with the purpose and agenda of the assignment; -agreeing upon approach for working with USAID; -developing a common understanding of the assignment’s objectives and outcomes; and -developing a preliminary draft outline of the expected report § Once the report is in the final draft format, the consultant will present the written findings to USAID/W, TBCTA and PAHO for comments. If deemed necessary, USAID may request an oral presentation. Interviews § USAID/W (CTOs, TB team leader and possibly other members of the TB team) § TBCTA PMU and staff of partner agencies involved in training activities § PAHO § Junior consultants ¾ Trainees of the various other TBCTA consultant training activities listed in Annex 2. § TFT members involved in the development of the training guidelines § Clients at the country level ¾ Supervisors of junior consultants VI. SCHEDULE The assignment will be conducted from April 18, 2005, through June 17, 2005. The following table is the projected schedule for implementing the methodology. It may be refined as a result of the planning meeting that will take place at the beginning of this assignment. 41 Table 1: Tentative Implementation Schedule Task Timeline 1. ASSIGNMENT START-UP 1.1 Review of Documents April 18-27 1.2 Team Planning Meeting (TPM) 1.2.1Conduct TPM (USAID/W) April 18 1.2.2Adjust Scope of Work April 19 1.2.3Obtain CTO approval of SOW April 20 1.3 Develop evaluation tools and guidelines 1.3.1Develop interview questions and analysis guidelines April 28-May 2 1.3.2Conduct USAID/W and other local interviews May 3-5 2. INTERVIEWS 2.1 Schedule interviews May 3-6 2.2 Conduct telephone interviews May 9-20 3. REPORT PREPARATION AND DEBRIEFING 3.1 Prepare and submit Draft #1 Executive Summary & full report to USAID/W and partners May 23-June 3 3.1.1Review of Draft #1 by USAID/W and partners June 10-15 3.2 Debrief USAID/W and TBCTA Members, separately June 13 3.3 Receive comments from USAID/W and TBCTA members June 15 3.4 Revise Draft #1 (becomes final version) June 17 3.5 Feedback to final version/approval by USAID June 17 3.6 Final report submitted to USAID June 17 TOTAL LOE 7 .5 .5 3 2 3 10 8 1 2 1 38 VII. DELIVERABLES AND CLOSE OUT Draft Executive Summary and Full Report and Briefing with USAID/W: An evaluation report with a summary and analysis of the findings, and recommendations for program improvements. The report should not exceed 30 pages with an executive summary of no more than 3 pages. Final Executive Summary and Full Report: After comments are received from USAID and partners on the first draft, the final Executive Summary and Full Report will be prepared incorporating the comments received from the review of the drafts. The final versions shall be submitted to USAID/W on diskette in Word format and pdf format. One hard copy will be provided. USAID/GH/HIDN will be responsible for distributing copies to the partners. VIII. TEAM COMPOSITION, QUALIFICATIONS, AND LEVEL OF EFFORT Human Resource Development Specialist: A human resource development expert with public health experience in developing and transitional countries. The person should have experience in conducting program evaluations. In addition, the person needs to have a good understanding of USAID operational, management and technical approaches. Finally, s/he should have strong English language skills as well as proficiency in Spanish. 42 The level of effort for the consultant is estimated and outlined in Section VI, Table 1, Tentative Implementation Schedule IX. LOGISTICS PATH will provide the following technical and logistical support: ƒ Identify and recruit consultant, and manage and support the person while on assignment. ƒ Provide administrative support for arranging any travel. ƒ Provide all expenses for the consultant. ƒ Provide support and editing services for the preparation of the final versions of the deliverables, if necessary. X. FUNDING This assignment will be funded by Global Funds earmarked for tuberculosis. 43 ANNEX 1 Junior Consultant Contact Information Name Organization Function Training Period Email and Phone Dr. Enrique PAHO Junior February 2005 – echegare@paho.org Echegaray Consultant December 2005 Dr. Vary Jacquet PAHO Junior March 2005 – jacquetv@paho.org Consultant December 2005 Lic. Claudia PAHO Junior February 2005 – monteroc@paho.org Montero Consultant December 2005 Dr. Matias PAHO Junior January 2004 – villatom@paho.org Humberto Villatoro Consultant November 2004 Reyes Dr. Andrés PAHO Junior January 2004 – andherna02@salud.gob.mex Hernandez Consultant November 2004 Dr. Alfonso PAHO Junior January 2004 – atenorio@iname.com Tenorio Consultant November 2004 Dr. Andrea Luna PAHO Junior February 2003 – marsana50@hotmail.com Consultant December 2004 aluna@ispch.cl Dr. David Zavala PAHO Junior Consultant February 2003 – December 2004 dzavala@ec-red.com Dr. Masoud Dara KNCV Senior Consultant October 2003 – October 2004 daram@kncvtbc.nl Joanne Sheppard WHO Senior August 2003-July sheppardj@who.int Consultant 2004 Dr. Nevin Wilson UNION Junior Consultant February 2004 – July 2005 nwilson@iuatld.org Dr. Netty Kamp KNCV Junior July 2003 – kampn@kncvtbc.nl consultant December 2004 Dr. Yelena WHO Junior March 2004 – yyu@euro.who.int Yurasova Consultant February 2005 Dr. Pangani UNION Junior March 2004 – pdhliwayo@iuatld.org Dhliwayo Consultant August 2005 Dr. Jean-Louis Union Junior February 2004 – jlabena@iuatld.org Abena Consultant July 2005 44 ANNEX 2 TBCTA Consultant Training Activities with Budget Partner Course Budget APA 5 KNCV Training of general consultants: KNCV office in Eastern Africa $80,000 WHO Training of TB consultants $105,423 WHO Training of TB consultants in DOTS Plus $63,935 ATS Increasing international TB consulting capacity in US $12,000 APA 4 ATS Increasing international TB consulting capacity in US $17,000 KNCV Training of TB consultants (Sondalo) $11,000 WHO Training of TB consultants (Sondalo) $97,254 APA 3 WHO Training senior HRD and general TB consultant for 11 months $216,508 KNCV Training 2 senior TB consultants for 18 months $224,000 The Union Training 2 senior TB consultants for 18 months $279,111 ATS Increasing international TB consulting capacity in US $49,000 WHO Technical Briefing $41,810 PAHO Fellows program budget is approximately $250,000 per year. 45 Attachment II: Documents Consulted Tuberculosis Coalition for Technical Assistance (TBCTA) Documents Training Course for TB Consultants Report, February, 2004 Report, Sondalo Global Ttraining, May 2004 Final report, Sondalo, November 2003 DatabasePart.GlobalSondalo Tuberculosis Coalition for Technical Assistance Mid-Term Evaluation, April 2004 Human Resource Development for TB Control. The role of TFT within TBCTA: A position paper, February 2004 KNCV Tuberculosis Foundation Junior Consultant Recruitment Ad Progress Report on Union Junior Consultants at the Paris Board Meeting TBCTA document, including the strategic framework and plan for collaboration with USAID Agenda, Junior Consultant Meeting, October 28, 2004 Minutes of the Meeting with "Junior Consultants," Paris, October, 1994 Guidelines on how to prepare a personal training plan for future TB consultants for TBCTA Personal Training Plan, Dr. Masoud Dara Personal Training Plan Template Increasing International TB Consulting Capacity in the US TBCTA Goals and Objectives Final report of the 3rd DOTS Expansion Working Group Meeting, 5–6 October 2002 Montreal, Canada Travel Report, Washington, April 2005 Pan American Health Organization (PAHO) Documents 2003 Documents: Programa de Adiestramiento en Tuberculosis de USAID - OPS 2003, PowerPoint presentation by Dr. Rodolfo Rodriguez, 2003 Tuberculosis y Situacion De Multidrogoresistencia, PowerPoint presentation by Dr. Andrea Luna Heine 2003 Tuberculosis y VIH en las Americas, Powerpoint presentation by Dr. David Zabala Rosas, 2003 TB Bulletin, Vol 6, No. 1, February 2003 TB Bulletin, Vol 6, No. 2, July 2003 TB Bulletin, Vol 6, No. 3, December 2003 2004 Documents: Informe de Residencia Programa de Adiestramiento en Tuberculosis, Dr. Alfonso Tenorio Gnecco, 2004 Workplan 2004, Dr. Andres Hernandez Workplan 2004, Dr. Alfonso Tenorio Workplan 2004, Dr. Matias Villatoro Programa de Adiestramiento en Tuberculosis USAID – PAHO, Alfonso Tenorio TB Bulletin, Vol 7, No 1, March 2004 TB Bulletin, Vol 7, No 2, August 2004 Brown bag lunch on TB in the Americas flyer Tuberculosis en pueblos Indigenas, PowerPoint presentation by Dr. Andres Hernandez, 2004 La TB en Centros Penitenciarios, un desafio para su control, Region de las Americas; PowerPoint presentation, Dr. Matias Villatoro, 2004 Las Reformas del Sector Salud, PowerPoint presentation by Dr. Alfonso Tenorio Gnecco, 2004 Epidemiological and operational situation of TB in prisons, Poster presentation by Dr. Matias Villatoro, 2003 Tuberculosis Fact Sheet, 2004 2005 Documents: Aide Memorie, Meeting for Selecting TB Fellows (2005) Program de Residencia de Tuberculosis - OPS Plan de Trabajo, Claudia Montero USAID/PAHO Training Program in Tuberculosis, Application Form USAID/PAHO Training Program in Tuberculosis, Terms of Reference 46 Terms of Reference USAID/PAHO Training Program in Tuberculosis Pan American Health Organization-World Health Organization 134th Session of the Executive Committee, Washington, D.C., USA, 21-25 June 2004 USAID/PATH Documents USAID/PATH Scope of Work GB Migliore Documents Documents Global Sondalo Courses 2003-2004, Database Training Course for TB Consultants, Report June 2004 Training Course for TB Consultants, Report October 2004 Training Course for TB Consultants Report February 2004 Sondalo Course Participants May24-5 June 2004 KNCV Fund Documents Personal Training Plan, Masoud Dara Briefing on Training Aspect for M. Dara Increasing International TB Consulting Capacity in the US Consultant list, KNCV, WHO, and the Union Overview Junior Consultants training, TBCTA Increasing International Tuberculosis Consulting Capacity in the United States Technical briefings for consultants Training Course for TB Consultants, Report October 2004 Training of Programme Managers for the Implementation of TB/HIV Collaborative Activities Report Personal Training Plan: Dr. Yelena Yurasova KNCV Advertisement for Junior Consultants The World Health Organization (WHO) Documents Training of TB consultants for missions to monitor and review DOTS Implementation in Countries Participant manual, Sondalo, final Evaluation of Participants, Sondalo, October 2004 Evaluation of Participants, Sondalo, May 2004 WHO Training Course for TB Consultants, 3-16 October 2004, Sondalo, Participants List Training Course for TB Consultants, Report, October 2004 The International Union Against Tuberculosis and Lung Disease (“The Union” or IUATLD) Documents Republic of Uganda National Tuberculosis and Leprosy Programme: Executive Action Document Biography of Dr Jean-Louis Abena-Foe CV, Nevin Charles Wilson CV, Panganai Dhliwayo On-the-Job Training Announcement for Two Junior International Tuberculosis Consultants from High and Intermediate Tuberculosis-Burden Countries Summary of activities, J. Labena Other Documents Bergstrom K. Training for Better TB Control: Human Resource Development for TB control - a Strategic Approach Within Country Support. Stop TB, World Health Organization, Geneva, Switzerland. The Task Force Training (TFT) of The Tuberculosis Coalition for Technical Assistance (TBCTA) World Health Organization, 2002 Figueroa-Munoz J. Palmer, K. Dal Poz, MR. Blanc, L. Bergström, K. and Raviglione, M. The health workforce crisis in TB control: a report from high-burden countries. Human Resources for Health, 2005; 3:2. International Tuberculosis Course Report, Arusha, Tanzania République de Madagascar, Programme National de Lutte contre la Tuberculose Applied Epidemiology for Operations Research in Tuberculosis Control, Paris Report Abstract-Form Paris 2004 Report: Advanced Course On AFB-Microscopy aAnd iIts External Quality Assurance (EQA) 2nd International Course on Management, Finance, and Logistics for TB Control, Course Evaluation Form 47 Report: International Course in Management, Finance and Logistics for TB Control Fanning, Anne, Draft Report Describing the Web-Based Course Funded by TBCTA Fanning, Anne, Web-Based Course Schedule Agenda, Junior Consultant Meeting, October 28, 2004 Personal Training Plan Human resource development for TB control: The roles of the TFT and other players within TBCTA, A position paper Junior Consultant Contact Information HIV and AIDS Conference, Zimbabwe: Taking Stock, Looking to the Future Minutes of the Meeting with “Junior Consultants,” Paris October Scope of Work: Evaluation of the Training Program for International Consultants Evaluation of a standardised recording tool for tuberculosis patients in Chitungwiza City Zimbabwe Tuberculosis Control Program Manual 48 Attachment III: Interview Instruments Questionnaire for Participants in the TBCTA Program to Train International Consultants in TB Control...50 Questionnaire for Participants in the PAHO Program to Train International Consultants in TB Control.....55 Questionnaire for Participants in WHO Courses to Train International Consulltants in TB Control............60 Questionnaire for Director of Junior Consultant Training Program..............................................................64 Questionnaire for Supervisor of Junior Consultant Training Program..........................................................65 Questionnaire for Mission Supervisor or Preceptor of Junior Consultant Training Program........................67 Questionnaire for PAHO Staff Implementing Junior Consulting Training Program ....................................68 49 ___________________________________________________________________ __________________________________________________________________________ Attachment III: Interview Instruments Questionnaire for Participants in the TBCTA Program to Train International Consultants in TB Control As part of a commitment to increase the pool of skilled technical assistance available to help TB programs, USAID has provided support to WHO, ATS, KNCV, The Union, and PAHO to train a group of TB consultants. As part of an evaluation of this program, we would very much appreciate your responses to the following questions. Your input will help us learn about your experiences in the training program, benefit from your suggestions for improvement, and understand how you are now applying the skills gained. The request for this evaluation comes from The U.S. Agency for International Development, Bureau of Global Health, Office of Health, Infectious Diseases and Nutrition. PATH, under the Global Health Task Order, has been asked to review the performance, impact, and lessons learned of the Training Program for Tuberculosis (TB) Consultants under the Tuberculosis Coalition for Technical Assistance (TBCTA) project and Pan American Health Organization (PAHO) grant. Your feedback will help improve the preparation of TB technical consultants. In responding to the questions, please provide as much detail as possible to facilitate our understanding and analysis of the training program. Thank you in advance for your time and effort. Participation Dates - Start ______________ Finish _______ Country of Origin _______ Job/Position when selected _________ Present Position ________ Selection and Preparation 1. How were you selected for this program?– Self ___ Workplace recommended ___ MOH ___ Other ____ 2. Did you provide TB consultation prior to participation in the program? Yes __ No __ 3. Were you/are you on leave from your position? Yes __ No __ 50 4. Did you discuss expectations for course participation with your workplace supervisor prior to taking this course? Was there an expectation that your responsibilities would change post￾training to include more consultation? Yes __ No __ Yes __ No __ Objectives 5. What personal goals motivated you to apply for this program? 6. Please rate your comfort level in international consulting before starting the program. High ____ Medium ___ Low ____ 7. In What technical or consulting areas did you feel you needed to develop more knowledge or skills to be an effective international consultant? 8. Were these areas addressed through your experience? Please elaborate: Yes __ No __ Personal Work plan 9. Please explain how your work program for the year was developed. 10. What were your learning objectives? 11. How did the staff of your training organization assist in developing the plan? Were you assigned an individual mentor or supervisor? Name of mentor: 12.Was a review conducted of your prior education and training your work experience your skills your personal ambitions Elaborate: Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 51 ____ 13.Did the plan include a mix of training activities and learning experiences? Yes __ No __ a. In your opinion was the mix balanced to meet your needs b. Was the plan effective Yes __ No __ c. Was the plan periodically reviewed and revised? Yes __ No __ Comment: Yes __ No __ 14.Were you satisfied with the work plan development process? Yes __ No __ Please elaborate and suggest any changes to the process: 15. Did you feel your work plan met your training needs? Yes __ No __ Elaborate: Training Courses 16. How many training courses did you attend? # _______ 17. In what way did the courses improve your skills – a. Was the range of available courses adequate to meet your needs? - comment Yes __ No __ b. Were you aware of other courses that would be more appropriate?- comment c. Were you able to apply for those courses? Yes __ No __ d. Did you experience any problems in registering for courses? Describe: Yes __ No __ Yes __ No __ 18. What could have been done differently to improve the course selection? Supervision/Mentorship 19. How often did you meet your mentor/supervisor? # What was the average length of those meetings? Length ___ 20. Did you feel the time and frequency of the meetings was adequate to Yes __ No __ meet your needs? Comment: 21. Was the time used to revisit and revise your plan? Yes __ No __ 22. Did you receive feedback during the meetings to strengthen your skills? Yes __ No __ 23. In what ways was the feedback helpful? 52 24. What are your suggestions for improving the mentorship and supervision provided by your training organization? Field Work 25.Did your mentor and or field mission leader help you to tailor your mission participation to address your learning needs? Yes __ No __ 26. How many missions did you participate in? - Did you feel the number and range was adequate to give you needed exposure and experience? # ____ Yes ___ No ___ 27. Did you gradually assume more responsibility in missions over time? 28.Were you satisfied with the missions chosen to supplement your learning? Yes __ No __ 29. Did your mentor/supervisor provide feedback to strengthen your field skills? Did your mentor/supervisor review your mission report? Yes __ No __ Yes __ No __ 30. Do you feel the missions helped to develop your skills and knowledge needed to be an effective consultant? What recommendations do you have to strengthen the learning value of the fieldwork? Yes __ No __ Self-Study 31. Did you and your mentor/supervisor collaboratively identify reading material for self-study? Did your mentor/supervisor discuss the material with you? Yes __ No __ Yes __ No __ 32. Was the material appropriate for addressing the identified training gaps? Why or why not? Yes __ No __ 33. What suggestions do you have, if any, for strengthening the self-study component of the training program? Current Situation 53 34. What is your present position? 35. How many TB consulting missions have you participated in since training? # ___ 36. Please list up to 5 clients that you have provided consulting services to since completion of the training program. Please also provide client contact information and the date of the consultancy: 37. Do you feel comfortable providing technical assistance in the following areas: 1. Evaluating TB programs 2. Writing mission reports 3. Preparing proposals 4. Advising on program and project implementation 5. Improving on program and project management 6. TB/HIV activities 7. DOTS Plus 8. Public Private Mix (PPM) 9. Operations Research 10. Drug management 11. Laboratory 12. Mobilizing the community 13. Advocacy at the national and local level 14. Integration of TB programs into the broader health system 15. Health financing 16. Other? Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 38. Please rate your comfort level in Consulting for WHO Consulting for other agencies Consulting directly for governments high medium low high medium low high medium low 39. Are there areas you feel should have been covered more rigorously in your consultant preparation? 40. Please share suggestions for improving the effectiveness of the training program. THANK YOU. 54 ___________________________________________________________________ __________________________________________________________________________ Questionnaire for Participants in the PAHO Program to Train International Consultants in TB Control As part of a commitment to increase the pool of skilled technical assistance available to help TB programs, USAID has provided support to WHO, ATS, KNCV, The Union, and PAHO to train a group of TB consultants. As part of an evaluation of this program, we would very much appreciate your responses to the following questions. Your input will help us learn about your experiences in the training program, benefit from your suggestions for improvement, and understand how you are now applying the skills gained. The request for this evaluation comes from The U.S. Agency for International Development, Bureau of Global Health, Office of Health, Infectious Diseases and Nutrition. PATH, under the Global Health Task Order, has been asked to review the performance, impact, and lessons learned of the Training Program for Tuberculosis (TB) Consultants under the Tuberculosis Coalition for Technical Assistance (TBCTA) project and Pan American Health Organization (PAHO) grant. Your feedback will help improve the preparation of TB technical consultants. In responding to the questions, please provide as much detail as possible to facilitate our understanding and analysis of the training program. Thank you in advance for your time and effort. Participation Dates - Start ______________ Finish _______ Country of Origin _______ Job/Position when selected _________ Present Position ________ Selection and Preparation 41.How were you selected for this program? Self ___ Workplace recommended ___ MOH ___ Other ____ 42.Did you provide TB consultation prior to participation in the program? Yes __ No __ 43. Were you/are you on leave from your position? Yes __ No __ 44. Did you discuss expectations for course participation with your workplace Yes __ No __ supervisor prior to taking this course? Was there an expectation that your responsibilities would change post￾training to include more consultation? Yes __ No __ 55 Objectives 45.What personal goals motivated you to apply for this program? 46.Please rate your comfort level in international consulting before starting the course. High ____ Medium ___ Low ____ 47.What technical or consulting areas did you feel you needed to develop more knowledge or skills to be an effective international consultant? 48.Were these areas addressed through your PAHO experience? Please elaborate: Yes __ No __ Personal Work plan 49. Please explain how your work program for the year was developed. 50. How did PAHO staff assist in developing the plan? Were you assigned an individual mentor or supervisor? 51.Was a review conducted of your prior education and training your work experience your personal ambitions Elaborate: Yes __ No __ Yes __ No __ Yes __ No __ 52.Did the plan include a mix of training activities and learning experiences? a. In your opinion was the mix balanced to meet your needs b. Was the plan effective c. Was the plan periodically reviewed and revised Comment: Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 53.In your plan did you include activities to address the following 1. Evaluating TB programs 2. Writing mission reports 3. Preparing proposals 4. Advising on program and project implementation 5. Improving on program and project management 6. TB/HIV activities 7. DOTS Plus 8. Public-Private Mix (PPM) 9. Operations research 10. Drug management 11. Laboratory 12. Mobilizing the community 13. Advocacy at the national and local level 14. Integration of TB programs into the broader health system Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 56 ____ 15. Health financing Yes __ No __ 16. Other? 54.Were you satisfied with the work plan development process? Yes __ No __ Please elaborate and suggest any changes to the process : 55. Did you feel your work plan met your training needs? Yes __ No __ Elaborate: Training Courses 56. How many training courses did you attend? # _______ 57. In what way did the courses improve your skills – a. Was the range of available courses adequate to meet your needs? Comment Yes __ No __ b. Were you aware of other courses that would be more appropriate? Comment c. Were you able to apply for those courses? d. Did you experience any problems in registering for courses? Yes __ No __ Yes __ No __ Yes __ No __ Describe: 58. In your opinion, in what areas were training courses lacking? Supervision/Mentorship 59. How often did you meet your mentor/supervisor? # What was the average length of those meetings? Length ___ 60. Did you feel the time and frequency of the meetings was adequate to Yes __ No __ meet your needs? Comment: 61. Was the time used to revisit and revise your plan? Yes __ No __ 62. Did you receive feedback during the meetings to strengthen your skills? Yes __ No __ 63. In what ways was the feedback helpful? 64. What are your suggestions for improving the mentorship and supervision provided by PAHO? Field Work 65.Did your mentor and or field mission leader help you to tailor your Yes __ No __ mission participation to address your learning needs? 57 66. How many missions did you participate in? - Did you feel the number and range was adequate to give you needed exposure and experience? # ____ Yes ___ No ___ 67. How did your role in the missions change over time? 68.Were you satisfied with the missions chosen to supplement your learning? Yes __ No __ 69. Did your mentor/supervisor provide feedback to strengthen your field skills? Did your mentor/supervisor review your mission report? Yes __ No __ Yes __ No __ 70. Do you feel the missions helped to develop your skills and knowledge needed to be an effective consultant? What recommendations do you have to strengthen the learning value of the fieldwork? Yes __ No __ Self-Study 71. Did you and your mentor/supervisor collaboratively identify reading material for self-study? Did your supervisor discuss the material with you? Yes __ No __ Yes __ No __ 72. Was the material appropriate for addressing the identified training gaps? Why or why not? Yes __ No __ 73. What suggestions do you have, if any, for strengthening the self-study component of the training program? Current Situation 74. What is your present position? 75. How many TB consulting missions have you participated in since training? # ___ 76. Do you feel comfortable providing technical assistance in the following areas: 1. Evaluating TB programs 2. Writing mission reports 3. Preparing proposals 4. Advising on program and project implementation 5. Improving on program and project management 6. TB/HIV activities 7. DOTS Plus 8. Public/Private Mix (PPM) Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 58 9. Operations research 10. Drug management 11. Laboratory 12. Mobilizing the community 13. Advocacy at the national and local level 14. Integration of TB programs into the broader health system 15. Health financing 16. Other? Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 77. Please rate your comfort level in: consulting for PAHO consulting for other agencies consulting directly for governments high medium low high medium low high medium low 78. Are there areas you feel should have been covered more rigorously in your consultant preparation? 79. Please share suggestions for improving the effectiveness of the training program. THANK YOU! 59 ___ __ Questionnaire for Participants in WHO Courses to Train International Consultants in TB Control Name _______Gender ________ Nationality______________ Date of Training _________________________ Selection & Preparation 1. How were you selected for this course? Self-selection Workplace recommended ___ Program recommended Other ______ 2. Did you provide TB consultation prior to participation in the course? If yes, please describe previous consulting experience: Yes ___ No ___ 3. Did you discuss expectations for course participation with your supervisor prior to taking this course? Yes ___ No ___ 4. Was there an expectation that your responsibilities would change post-training to include more consultation? Yes ___ No ___ Course Evaluation 5. Were your training needs evaluated at the start of the course? How? Yes ___ No ___ 6. If so, was the information used to address your personal needs? How? Yes ___ No ___ 7. You came in with a set of skills. What did you expect from this course in terms of added knowledge and skill? Please be specific. 8. Did the course meet your objectives? Please explain why or why not. Yes ___ No ___ 9. What part of the training contributed most to building your skills? 10. What parts of the training were not helpful? 11. Did you find the course objectives relevant and achievable? Yes ___ No ___ 60 12. How helpful was the written material you were provided at the course to improving your knowledge and skills? Please elaborate on your rating: Not helpful ___ Helpful ____ Very helpful ____ 13. Did you find the instructors well prepared and informative? Yes ___ No ___ 14. Did you find the role-playing to be an effective learning methodology? Please elaborate: Not effective_____ Effective_____ Very effective______ 15. Do you think there was a good balance between lectures, role￾playing, teamwork, practical skill building (example: reading smears), and report writing in the activities of the course? Please elaborate: Yes ___ No ___ 16. Did the course contribute to improving your general consulting skills (example: listening, probing, team work, team leadership, writing, presentation)? Please elaborate: Yes ___ No ___ 17. Did you feel the number of participants was satisfactory for both productive group learning and individual attention? How would you change it? Yes ___ No ___ Smaller ___ Larger ___ 18. Was language an issue in understanding or participating? Yes ___ No ___ 19. Do you feel the course should continue in English? Yes ___ No ___ 20. Do you feel any change should be made in the international mix of participants or setting? Elaborate: Mix: Yes ___ No ___ Setting: Yes ___ No ___ 20. Did you feel the length of the course was adequate for the amount of material covered? If you were to change it would you make it longer or shorter? Yes ___ No ___ Shorter ___ Longer ___ Same ___ 21. What other improvements would you suggest for the course structure? 22. Did the course adequately prepare you to write the mission report? Yes ___ No ___ 61 23. Was language a problem in finalizing the report? Yes ___ No ___ 24. Did you find the feedback you received on the report helpful to improve your performance? How? Yes ___ No ___ 25. Did you find the feedback on your course participation and next steps helpful for developing you as a consultant? Yes ___ No ___ 26. How did you use the feedback to further your professional development? Present Situation 27. What is your current position? Are you with the same organization as before training? Position________ Yes ___ No ___ 27. If you are still with the same organization, do you have the opportunity to use your new skills in your present position? Yes ___ No ___ 28. If your position changed, do you provide TB program consultation in your present position? Yes ___ No ___ 30. How often have you provided TB control consultation since your training and for what agencies and countries? Please list the agencies, countries, and dates of consultancies: #___ # In country ___ # Internationally ___ 31. Looking back, do you feel the case studies and exercises mirrored what you found on actual missions? Elaborate: Yes ___ No ___ 32. What do you think was missing from the course that would have helped you to be a more effective consultant? 33. What type of support would you have liked post-training? 34. Have you had any contact with WHO or other agencies that have access to the consultant database requesting your participation in mission? WHO: Yes ___ No ___ Other: 35. How would you rate your present comfort level in: a. Evaluating TB control programs b. Strengthening program implementation c. Proposal writing d. Working for a range of clients Low Medium High Low Medium High Low Medium High Low Medium High 62 30. Please share suggestions for improvement or provide other feedback. THANK YOU! 63 Questionnaire for Director of Junior Consultant Training Program Recruitment and Selection 1. How do you recruit candidates for the junior training program? 2. How many candidates can your organization accommodate annually? a. Is this number based on i. Supervisory capacity ii. Financial limitations iii. Other 3. Where is your candidate physically based? a. What are the advantages and disadvantages of this set-up? 4. How many applications have you received? 5. What criteria do you use to select participants from the proposed candidates? a. Education b. Prior training c. Experience – specialized skills d. Organization needs e. High burden country needs 6. Do you feel the length of time is adequate for developing needed skills? Structure 7. How do you orient staff to their roles as supervisor? 8. What process do you use to match the trainee to a supervisor? 9. What material does the supervisor receive before developing the personalized training plan? (e.g., the application, interviews, review of past training, education and work experience) 10. Are supervisors allotted adequate work times for taking on supervisory roles on top of their normal responsibilities? Communication 11. Is the PMU responsible for keeping training information up to date? a. How are you informed of new and appropriate courses? b. How are you informed of missions from other organizations appropriate for “trainees”? 12. Are you informed about potential job openings for “graduates”? 13. How do you promote candidates for potential jobs? 64 Questionnaire for Supervisor of Junior Consultant Training Program Preparation 1. How were you prepared for your role as supervisor for this candidate? a. Did you receive the guidelines for developing a personal training plan? b. Were you oriented to the responsibilities of the supervisor? 2. Were you given adequate time from your normal tasks to take on the supervisory role? Have you mentored trainees before? 3. What knowledge of this candidate did you have before developing the personalized training plan (PTP)? (e.g., the application, interviews, review of past training, education and work experience) 4. Did you review the contents of the courses included in the guidelines as potential training opportunities? Did you have a list of upcoming missions/purpose before plan development? a. How are you informed about missions from other organizations that the trainee could attend? b. How are you informed about relevant new courses? 5. Did you review the reading list? a. Is there a mechanism for identifying new reading material for self-study? Planning 6. Describe the process for developing the personalized training plan. a. How long did it take to develop the initial plan? b. How did you assess the trainee’s performance level on the competencies for TB consultants (included in the guidelines for preparing personal training plans)? Do you find these competencies on target and helpful? c. Did you mutually agree on areas of need? 7. Were the training options adequate for the trainee’s need? Did you or the trainee suggest alternative courses or reading materials? 8. Were you informed about missions from other organizations that could be included in the plan? 9. Was the PTP format adequate for developing an individualized plan? a. Do you have any recommendations for change? Supervision 10. How often did you review the trainee’s progress? a. Were supervisory sessions regularly scheduled or ad hoc? b. As a result of these meetings was the plan revised? 11. How would you strengthen the role of supervision? 12. How do you ensure that topical issues are addressed through the training program, such as consulting on: a. Integration of HIV/TB programming. b. Innovative use of staff such as lay workers or volunteers. c. Other emerging issues. 65 Coordination 13. What information did you give the field mission preceptor about the candidate’s training needs? 14. How did you ensure that the candidate’s training needs were recognized and addressed and he/she was incrementally groomed for independent consultation? 15. Did the preceptor submit a report? Did you review the trainee’s mission participation and report with the preceptor? Evaluation 16. Do you think the length of the training period is adequate to develop a TB consultant with generalized skill? 17. Do you think the selection process adequately identifies trainees with potential to be a TB consultant? a. Are there other criteria you would like considered when screening candidates? 18. Do you feel the range of training opportunities adequately addresses the competencies required: a. Writing mission reports b. Preparing proposals c. Advising on program and project implementation d. Improving on program and project management e. TB/HIV activities f. DOTS Plus g. Public-private Mix (PPM) h. Operations research i. Drug management j. Laboratory k. Mobilizing the community l. Advocacy at the national and local level m. Integration of TB Programs into the broader health system n. Health financing o. Other? 19. Do you feel the training was experiential enough to give the candidate the skill required to perform as a senior consultant? 20. What changes would you suggest to strengthen the overall training process? 21. How do you assess the candidate’s readiness to be a consultant? 66 Questionnaire for Mission Supervisor or Preceptor of Junior Consultant Training Program 1. What information did you receive about the candidate consultant before participation in your mission? 2. What was the learning objective of the trainee’s participation in your mission? 3. What tasks did the trainee take on? 4. Did you feel the trainee had adequate preparation for her/his role? 5. Did you review the candidate’s mission report? 6. Did you provide feedback to the candidate? 7. Did you provide an assessment to the trainee’s supervisor? 8. Did the trainee participate in more than one mission with you? a. How did you work toward giving the candidate greater independence? 9. Did you feel the candidate was capable of taking on independent missions after his/her experience? 10. Please indicate how you would strengthen the: a. Communication between the supervisor and yourself b. Pre-mission preparation c. Post-mission training d. Mission participation . 67 Questionnaire for PAHO Staff Implementing Junior Consulting Training Program Position _______________ Background 1. What is the goal of the junior consultation training program? 2. How do you assess whether the candidate is appropriate? 3. Does the application provide enough detail about the applicant’s potential as well as training needs Yes __ No __ 4. Is there any information you feel should be included to better match the applicant with a supervisor/mentor? Personal Workplan 5. Do you use the application to identify gap areas for the trainees? Yes __ No __ 6. What is the format for developing a personalized work plan for the trainee? 7. Is the plan collaboratively developed between the supervisor and trainee? Yes __ No __ 8. Is there a format for developing the plan? Does it include: ƒ Identified Skill and Knowledge Needs ƒ Personal and Program objectives ƒ Activities to meet the needs ƒ Evaluation ƒ Next steps Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ Yes __ No __ 9. How are the trainee’s activities determined? What is compulsory? What other activities are available to individualize the content? Probe: classroom, field work, self-study, shadow a senior consultant, etc. 10. Does the trainee meet with the supervisor to review the relevance of the activities and assess progress? Yes __ No 11. Is the meeting used to revise the work plan if necessary? Yes __ No 12. How often does the supervisory meeting occur and for how long? # ___ Length______ 13. Is it scheduled on a regular basis or ad hoc? 14. Is supervision of trainees included in PAHO supervisor job descriptions or is this seen as an added task? 15. Are PAHO staff briefed on their role as supervisors? Yes __ No 68 ____ Mission Preparation 16. What is the purpose of mission participation? 17. How are the missions arranged to meet individual knowledge and skill needs? Probe: ƒ Do supervisors attend the mission? ƒ Is the role of the consultant carefully structured and communicated to the team leader or other consultants? ƒ Is there a written evaluation by a mission member? ƒ Who reviews the consultant’s report? ƒ Who provides feedback to the consultant? Yes __ No Yes __ No Yes __ No Yes __ No Yes __ No Course Selection 18. How is course selection matched to individual needs How are they selected – predetermined, by trainee, by PAHO? Predetermined___ Trainee selected __ PAHO selected __ Other 19. Is the trainee expected to write an evaluation of course appropriateness and relevance? Yes __ No 20. How is self-study linked to individual need? ƒ How are the materials selected? ƒ Is the trainee expected to discuss the reading with the supervisor? 21. Are there other activities designed to improve skill in areas such as planning, program implementation, or evaluation? Describe: Post-training 22. How do you evaluate a candidate’s readiness to take on individual missions? Describe evaluation process: 23. How is the graduate’s availability communicated to the region and to whom? 24. Does the graduate return to his/her workplace? Yes __ No __ 25. Is an agreement reached with the workplace to allow for leave for consultation? Yes __ No __ 26. Does his/her absence affect the workplace? Describe: 27. Do you have any suggestions for strengthening the individualized learning program? Describe: 69