EVALUATION AND OPTIONS FOR DEVELOPMENT TRAINING 2 Lessons Learned from an Ambitious Training Management Experiment and Options for Promoting Global Performance Improvement for U.S. Agency for International Development Egypt Submitted to USAID/EGYPT Office Of Human Development And Democracy Cairo, Egypt By AGUIRRE INTERNATIONAL 1156 15th Street, N.W Suite 1000 Washington, D.C. 20005 Andrew Gilboy Barbara Hunt Jeanne Moulton Gerald Wein This project assessment was conducted by Aguirre International under GEM Task Order #802 with USAID/Egypt, Contract Number FAO-1-99-00-00010-00, dated December 26, 2000. The findings, conclusions, and recommendations are those of its authors and do not necessarily reflect the views and/or policies of USAID/Egypt. March, 2001 Evaluation and Options Development for DT2 Page i USAID/Egypt Aguirre International Evaluation and Options Development for DT2 Table of Contents List of Annexes............................................................................................................................... ii List of Abbreviations .....................................................................................................................iii Introduction.................................................................................................................................... iv Executive Summary........................................................................................................................ v I. Purpose and Methodology .................................................................................................. 1 A. Background ............................................................................................................................1 B. Purpose of the Report.............................................................................................................3 C. Methodology and Survey Population.....................................................................................5 D. Organization of the Report.....................................................................................................7 II. Findings............................................................................................................................... 9 A. Principal Issues ......................................................................................................................9 1. Assessing training needs................................................................................................................9 2. Planning and processing training.................................................................................................13 a. Training Design & Specifications ........................................................................................13 b. Procurement .........................................................................................................................14 c. Processing overseas participants ..........................................................................................17 3. Monitoring, follow-on and evaluation.........................................................................................18 4. Training data collection and management...................................................................................21 5. Contractor management...............................................................................................................23 6. Addressing USAID's Evolving Needs.........................................................................................25 B. Funding Mechanisms ...........................................................................................................27 III. Key Recommendations and Options................................................................................. 31 A. Balance Sheet on DT2..........................................................................................................31 B. Funding ................................................................................................................................34 C. Recommendations Concerning the Principal Management Issues.......................................37 ANNEXES Page ii Evaluation and Options Development for DT2 Aguirre International USAID/Egypt List of Annexes A. Case Studies and Criteria for Selection B. Training Needs Assessments – Supplementary Information C. Quick Reference to Results Oriented Training: Best Practices Abridged for USAID Stra￾tegic Objective Teams D. USAID/Egypt Options Meeting Powerpoint Presentation E. Scope of Work and Aguirre Team Work Plan F. Interview Questionnaire G. List of Persons Interviewed by Affiliation H. Documents Reviewed Evaluation and Options Development for DT2 Page iii USAID/Egypt Aguirre International List of Abbreviations ADS Automated Directives System (of USAID) ADS 253 Provides USAID’s policies and procedures governing training ALIGU American Language Institute/Georgetown University CEPA Communicative English Proficiency Assessment COTR USAID Contract Officer’s Technical Representative COP Chief of Party (the DT2 Contractor’s head person in Egypt) CTO Cognizant Technical Officer DT2 or DT II Development Training II Project DT3 Development Training III, the proposed follow-on project to DT2 ELT English language testing ELTT English Language Teaching and Training Program EPI Egyptian partner institution EPT English Proficiency Test FOE Faculty of Education HDD/ET Office of Human Development and Democracy (of USAID/Egypt) ICT In-country Training IELP-II Integrated English Language Program II IIE Institute of International Education IQC Indefinite quantity contract IR Intermediate result KSA Knowledge, skills and attitudes OFE Orientation, follow-up and evaluation OST Observation and Study Tour OYB Operational Year Budget PIR Performance Improvement Result SO Strategic Objective SOT Strategic Objective Team SOW Scope of work TA Technical assistance TAC Technical Assistance Contractor TGP Training for Global Performance – the proposed name for "DT3" TNA Training needs assessment TOT Training of Trainers TP Training plan TPA Training partnership agreement TraiNet The training management data system used by USAID Missions WU Work unit Page iv Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Introduction The research for this report was undertaken in January 2001 in Washington, D.C. and from Feb￾ruary 3 to March 9, 2001 in Cairo, Egypt. The team was comprised of the following consultants brought together by Aguirre International: Andrew C. Gilboy, a human resources and evaluation specialist (Team Leader) Barbara Hunt, an education specialist with extensive evaluation and testing experience Jeanne Moulton, an evaluation/project management expert focussed on education and training Gerald Wein, an economist and former senior USAID official Substantive and logistic backstopping was provided by: Roger Rasnake, evaluation specialist and project manager at Aguirre International. The Team wishes to express its appreciation for all the assistance made available by the Office of Human Development and Democracy at USAID/Egypt, in particular Jim Van Den Bos, the CTO for DT2, and Fatma Naguib, Training Assistant, both in the Education and Training Section, as well as Steve Brent, Associate Director for HDD and Sally Patton, Chief of Human Develop￾ment. Many other officers at USAID devoted considerable time answering the Team's requests, such as Gary Kinney, David McCloud, Salwa Nashed, Adel Halim, Anthony Vance and Joan Larcum. The research could never have been conducted without the cooperation and active participation of the staff at IIE's office in Cairo, in particular Melanie Sanders-Smith, Chief of Party, Faith Galetshoge, Director of Training and the entire staff. Everyone graciously gave of their time and located files and information essential to the findings and recommendations made by the Team. Finally, the Team expresses appreciation to the TA Contractors and Egyptian Institutional Part￾ners, too many to mention individually, who devoted a large amount of their time to answering the Team's many questions and even inviting Team members to attend in in-country training events. Evaluation and Options Development for DT2 Page v USAID/Egypt Aguirre International Executive Summary This summary extracts the principal findings, key recommendations and options put forward in this report in order to present them succinctly with a minimum of text for the busy reader. If ad￾ditional clarification or justification is desired on any point, the reader is encouraged to consult the detailed Table of Contents and refer to the in-depth discussion in the report. Terminology: Throughout the report, the term "DT2" refers to the present USAID/Egypt activity begun in 1997 that centralized the management of Mission-funded participant training, and a significant portion of in-country training, under a single funding source. The activity is managed by the DT2 Contractor, the Institute for International Education (IIE), selected in 1997 through competitive procurement. A distinction is made in this report between DT2 and IIE. When re￾ferring to the DT2 design features, funding mechanism or overall program, the term "DT2" is used; when referring to specifics about contractor management of the DT2 activity, or the con￾tractor's procedures, performance or approach, the term "IIE" is used. To avoid confusion when referring to mechanism through which USAID/Egypt will manage its training from 2003 onward, which has been referred to as "DT3," the authors of this report chose a new activity name, Training for Global Performance, or "TGP." This new activity name is a proposal only and was not the subject of extended discussions with USAID staff. Background. The original justification for centralized training management rested on the desire to provide orientation, follow-on and evaluation for USAID/Egypt's participant training program. However, the eventual Project Paper (issued on August 20, 1995) included tasks and responsi￾bilities considerably beyond ensuring that these important elements of training management were properly conducted. In particular, Best Practices for "results-oriented" training were being de￾veloped in other areas of USAID and their principles and techniques were incorporated into DT2. Likewise, reengineering was getting into full swing and influenced the DT2 design in sig￾nificant ways. The DT2 design therefore grew to embrace modern training principles and approaches in addi￾tion to mandating that "Best Practices" be applied to all Mission-funded participant training. To reinforce the message, DT2 was also charged with managing selected in-country training pro￾grams where contractors might not have a comparative advantage. The DT2 project therefore aimed to fulfill the following objectives: a) correct non-compliance in implementation of overseas participant training, primarily with the existing Handbook 10 regu￾lations, b) streamline training management by centralizing funding to ease in-house manage￾ment, and c) ensure that training contributed to Mission SO/RP results All of the attention in the early design years was on centralized management, not centralized funding. The idea of centralized funding apparently surfaced as a convenience and was not a Page vi Evaluation and Options Development for DT2 Aguirre International USAID/Egypt feature in the original design articulated in internal memoranda. Moreover, a critical assumption upon which DT2's success rested, in terms of its ability to introduce Best Practices and adhere to proper training management, was that the level of USAID reengineering would be sufficient to enable the SO/RP teams to determine training needs in collaboration with their TA and Egyptian partners. Purpose of the report: To "develop options, based on an evaluation of DT2, for the management of training, participant and in-country, after the end of DT2 in 2003 and conduct a full-mission Training Options Meeting to consider evaluation findings, options on training management is￾sues, and strategies to improve Strategic Objective Teams' participatory role in integrating train￾ing into results package activities." Methodology Used. Structured interviews of 83 individuals were conducted using primarily open-ended questions. The survey population was drawn from case studies selected by the team that would reveal the dynamics of training management as seen from within a single training event. The 4-person Team interviewed 32 USAID/Egypt officials from all sectors, 38 TA Con￾tractor personnel, managers from 4 training providers and 9 participants. Since training impact per se was not the focus of the report, few participants were consulted directly. Findings Overall On the basis of its review, the Team concluded that DT2 a) has succeeded in bringing the man￾agement of training into compliance with USAID regulations; b) has made progress in stream￾lining training, although further progress is desirable; and c) has not greatly influenced the de￾gree to which USAID training is re-engineered or adheres to both the rule and the meaning of Best Practices. With regard to compliance to USAID’s training rules and regulations (ADS 253), centralizing participant training with an outside contractor has resolved most of the problems cited in audits and memos in the early 1990s. This positive result is due largely to IIE's careful attention to compliance issues and its close working relationship with USAID/Egypt staff directly responsi￾ble for training procedures. In its initial two years DT2 procedures were widely perceived as obstacles, and the pace of Mis￾sion-supported training declined. The manner in which DT2 assessed training needs, a principal source of the delay and annoyance during that period, has since been adjusted. These efforts have been quite successful, and DT2 is clearly much more “user-friendly.” IIE’s success in this regard is reflected in the greatly expanded numbers of Egyptians trained through DT2 during the past year. Measuring DT2’s contribution to promoting Best Practices is problematic. The initial contractor strategy included rigorous training needs assessments and aggressive programs to promote other similar practices, most of which generated anger and resentment in the very audiences whose Evaluation and Options Development for DT2 Page vii USAID/Egypt Aguirre International level of appreciation and commitment to Best Practices DT2 was supposed to advance. Cur￾rently, needs assessments and development of training specifications are largely in the hands of TA Contractors, and to a lesser extent USAID technical staff. In sum, the success that IIE had in shifting its focus over the past two years to meet USAID’s demand to facilitate and streamline training has limited the extent to which it provides professional advice to promote re-engineering and Best Practices. Assessing Training Needs • Understanding that the process of assessing training needs was an integral part of reengineering and part and parcel of Best Practices, IIE sought to apply the process with vigor from the beginning as called for in its contract and in the DT2 design. • The TNA models used by IIE in the early period were too lengthy, costly and in some in￾stances, were isolated from stakeholder input • Training programs were delayed as a result of the TNA "requirement" and resulted in signifi￾cant USAID training pipeline build-up • IIE data suggest no association between a lengthy and costly TNA and the results obtained as indicated in subsequent IIE on-site evaluations • IIE responded to the lack of support for the type of TNAs conducted in the early years by eliminating this requirement and reallocating resources to implementation • New ways to analyze performance gaps, such as using rapid appraisal techniques, have not been tried. Planning and processing training .… training design and specifications: • A three-year training plan, and even an annual plan, containing sufficient detail, lacked the flexibility that some SO/RP teams required to address actual training needs. • IIE has provided assistance in developing training plans and specifications for a wide variety of training activities. In many instances the SO/RP teams and TA Contractors have been satisfied with IIE's services. • Despite the significant improvements made by IIE in articulating training plans and program specifications over the last two years, more needs to be done to simplify the process. Page viii Evaluation and Options Development for DT2 Aguirre International USAID/Egypt .… procurement: • Most users are now satisfied with DT2 services and refer to problems in the past tense. The serious problems highlighted in the 1999 evaluation have been largely resolved • In-country procurement of courses has produced the unexpected advantage of strengthening the planning and management capacity of some local training firms • Though in most cases USAID/Egypt and its TA Contractors have been satisfied with IIE's procurement of short training programs, difficulties have emerged in the procurement of tai￾lored training programs for the private sector. • On several occasions, USAID/Egypt has asked IIE to support the design of large training programs and even to administer them, as a kind of surrogate "TA Contractor." These activi￾ties have diverted IIE staff resources for project management tasks that were not factored into the IIE contract level of effort. ….processing overseas participants: • Generally efficient and satisfactory services are provided by IIE, in contrast to the pre-DT2 years when no systematic and comprehensive pre-departure orientations were being con￾ducted by the TA Contractors • Problems arise principally due to "special" cases – private-sector participants travelling on partially-sponsored travel or high-ranking civil servants • IIE and USAID/Egypt have collaborated closely in trying to make Mission regulations more flexible and efficient; the new DT2 User's Guide has helped Monitoring, follow-on and evaluation: • The overall process for Monitoring, Follow-On and Evaluation is flexible, allowing for variation in whether IIE or a stakeholder provides the services • IIE's monitoring of participants while they are in overseas programs works well and is occa￾sionally supplemented by a TA Contractor's contacts or systems • IIE's follow-on programs are innovative and involve seminars, workshops and a $1,000 grant for participant use to further learning; however, the administration of the participant grant program is not always well-understood by participants or TA Contractors • IIE provides no follow-on services in the workplace, although in some cases the TA Con￾tractors may be providing such services Evaluation and Options Development for DT2 Page ix USAID/Egypt Aguirre International • IIE's evaluation system has been significantly streamlined since January 1999 and resources diverted to more urgent needs, such as implementing training Training data collection and management: • TraiNet collects participant training data of use to USAID/Washington and to a limited de￾gree, to USAID/Egypt • Participant data is believed to be accurate in view of the fact that all but a small number of overseas participants are processed by DT2/IIE • In-country data is believed to be incomplete because so much is implemented by TA Con￾tractors who do not submit the data to IIE; • TraiNet provides little value added in terms of data tracking and reporting, either to the TA Contractors or to USAID Egypt; most contractors, including IIE, have developed a custom￾ized data management system that duplicates and supplements TraiNet Contractor management: • IIE has made significant progress in streamlining and rationalizing all of its implementing functions • There has been continuing confusion about the division of responsibilities between IIE and the TA Contractors regarding in-country training and some other areas • In order to be more customer oriented, DT2 has moved to a “reactive” stance in which it is not actively emphasizing implementation of Best Practices Addressing USAID's evolving needs: • DT2 has enhanced the Mission's ability to respond to new initiatives, because it provides a central fund which can be reallocated as needs change. At the same time, the funds that DT2 has available to manage training (its "core") are fixed. • IIE staff have responded fully to Mission requests to manage the new initiatives, although the management intensity of some new initiatives has meant that the staff has been able to pay less attention to regular tasks outlined in its scope of work. Page x Evaluation and Options Development for DT2 Aguirre International USAID/Egypt • No concerted effort has yet been made by IIE or the Mission to build and sustain the capacity of Egyptian organizations to manage training as DT2 has done, which may pose problems as USAID's program decreases. • Under DT2 the Mission has improved dramatically its compliance with Agency guidelines regarding the inclusion of women in training programs. Over 30 percent of all participants in overseas programs – both short- and long-term – are women. • Despite its progress toward including women in training, the Mission will need to set a more ambitious target, possibly at 50 percent, for women overseas training participation, • Long-term academic training has decreased and if the Mission anticipates a significant change in these programs, TGP will need to develop transparent and well-publicized policies to recruit, and possibly even assist through remedial work and test preparation, women for university placement. Funding Mechanisms: ….advantages of a centralized system: • Because SO/RP teams and TA Contractors did not feel much ownership in their DT2 alloca￾tions, it has been relatively easy to reprogram funds to satisfy new Mission initiatives and the needs of RP teams that use up their allocations and make the case for more. ….disadvantages of a centralized system: • The DT2 funding mechanisms sets up a second resource allocation system that to some ex￾tent distorted resource allocation, favoring training over other inputs. Those receiving these allocations and responsible for seeing to their effective use had less flexibility in using them than they do with other funds. They often lack a sense of ownership over these resources and seem to put less care into their effective use. Key Recommendations and Options Funding The key funding issue for TGP is whether to continue with DT2’s system of centralized training resources, or, alternatively, to allocate those resources to SO/RP teams so that they might decide how much to spend on training. The Team recommends that USAID/Egypt decide to decentral￾ize the funding of its centralized training management mechanism, TGP, taking into account the advantages and disadvantages of the DT2 experience with a central training fund. The Team recommends that the Mission revert to the more traditional decentralized approach when it de￾signs TGP. Decentralization can be expected to yield benefits in the following area: Evaluation and Options Development for DT2 Page xi USAID/Egypt Aguirre International • Promoting administrative efficiencies and cost control • Streamlining the current duplicative and confusing allocation processes • Clarifying roles and responsibilities • Focusing activities of USAID training staff • Enhancing program efficiency and making better use of available resources by having RP teams, TA Contractors and grantees allocating resources to best achieve RPs. • Integrating TA and training The implementation of the decentralized funding mechanism will be similar to what is done in many other USAID Missions. A Mission selects a central contractor from the pre-competed central IQC (previously Global Training for Development, which will be replaced in June 2001 by START) through a mini-competition based on the TGP Scope of Work. That contractor opens a Cairo office to manage training as described in the SOW. That operation is entirely un￾der the supervision and contracting authority of USAID/Egypt. The contract is funded by each SO through SO-based Task Orders. Each Task Order can be negotiated with the contractor and includes the contractor support costs needed to implement the training described in the Task Or￾der. If the Mission wanted to fund some of the TGP core costs for certain cross-cutting activities or targets of opportunity training not directly related to an individual SO, it could issue a Task Or￾der directly. Or, were the Mission to fund a cross-cutting long-term academic training program, it could issue a specific Task Order for that program without going through an SO. Alterna￾tively, it could issue a single Task Order with multiple SO funding. All of the above methods have been used by other Missions with the GTD contract and would be possible with the new START mechanism. Recommendations Concerning the Principal Management Issues The management Options available to the Mission in designing TGP are similar to the options available to the Mission in 1993, with some changes: (1) Centralized management of most training funded by USAID/Egypt implemented by an outside contractor, as has been done under DT2 (2) Decentralized management of all training to TA Contractors, reverting back to the previ￾ous system but without heavy involvement of a USAID Training Office (3) A mixture of centralized management for certain types of training and decentralized for other types. The Team strongly recommends Option (1) where USAID/Egypt continues with the centralized management by a U.S.-based contractor of most of its training programs, whether in-country or Page xii Evaluation and Options Development for DT2 Aguirre International USAID/Egypt overseas. The DT2 experiment has been largely successful which would not argue for reverting to a system confiding important administrative decisions either to sector-specific TA Contrac￾tors, some without training management capacity, or to the USAID Mission itself, which cannot increase its staff or management burdens realistically in the current reengineered environment. Option (2) cannot be considered due to the inability to ensure standardized approaches to training processes and minimal requirements to ensure Best Practices with so many independent con￾tractors each handling training differently. The management of training of the magnitude of that found in the diverse portfolio of USAID/Egypt is best left to organizations with expertise in han￾dling large numbers of individuals, partner institutions and training providers. Option (3), if clarified carefully, could be considered. However, in selecting Option (1), the Team would include in its design prescriptions for TGP the provision that TA Contractors have flexibility in determining which organization (that is, the TA Contractor or the TGP Contractor) would handle designated management tasks and responsibilities. Option (3) would imply a pre￾emptive assignment of management tasks to either a central contractor or a sector contractor. Another dissuasive argument is that increasing amounts of training are cross-sectoral, such as "NGO management strengthening," "financial management," or "computer programming." Assessing Training Needs • Immediate: In light of the shift over the past two years from promulgating Best Practices by rigorously applying administrative procedures (e.g., TNAs) to managing training outputs, IIE and the CTO should review the human and financial resources available in DT2 to consider a more aggressive role as a catalyst for improving the way organizational needs are diagnosed. • Future: TGP should have the mandate and resources to provide high-quality technical and practical advice to SO/RP teams, TA Contractors, Egyptian consultant/trainers and partner institutions to promote the widespread use of contemporary organizational performance analyses by Egyptian firms, NGOs and government entities. Future training sponsored by USAID/Egypt must move from implementation to performance improvement with increasing participation and support from leading Egyptian training providers. The TGP mechanism of￾fers an ideal venue for this level of targeted assistance. Since the initial demand for such as￾sistance may not be high, the "Performance Improvement Advisor" working from the TGP Contractor would be responsible for creating demand for human performance technology that is adaptable, cost-effective and appropriate to the Egyptian context. Planning and Processing Training • Immediate: USAID/Egypt should allow the TA Contractor and IIE to amend the Training Plan as needed without specific Mission approval, as long as the new budget does not exceed what was previously approved. This change would help speed up training starts and put the responsibility squarely on the DT2 and TA Contractors to justify the changes made. USAID's Evaluation and Options Development for DT2 Page xiii USAID/Egypt Aguirre International relationship should evolve from "gatekeeper" and "watchdog" over training funds to "results￾monitor" of training • Future: To the extent permitted by Agency-wide guidelines (ADS-253), the USAID/- Egypt/HDD should continue its efforts to streamline travel regulations and to revise waivers to be in line with Mission programmatic objectives. Monitoring, Follow-on and Evaluation • Immediate: Follow-on in the workplace should be provided, at a minimum, for all long-term training, both in-country and overseas, and selected short-term U.S. programs. • Future: Based on the experience of DT2 over the next two years, the TGP design should clarify the type of training evaluation needed in order to support organizational performance improvement rather than be limited to fulfilling an administrative requirement. Contractor Management ….Immediate: • Begin the shift from a strategy of quality assurance to an advisory strategy by promoting better understanding of Best Practices among customers (periodic workshops, information dissemination, etc.) • Give the DT2 Contractor more flexibility to adjust its staff levels and capacities according to the diverse types of training requests from TA Contractors and SO/RP teams. • Consider transferring the oversight responsibility for DT2 to the Program Office which is a neutral location better suited to oversee multiple funding to a central contract by different SOs (a solution used by other Missions). The new CTO could work out of the Program Of￾fice in managing DT2 for all SOs. ….Future: • Design TGP to provide a mix of services that includes processing participants for training as well as enhancing the context for institutional performance improvement. • Include as one of TGP's objectives the strengthening of organizational capacity of Egyptian training providers Page xiv Evaluation and Options Development for DT2 Aguirre International USAID/Egypt • When TA Contractors or SO/RP teams choose to use TGP's services to procure tailored training programs, USAID/Egypt should have the flexibility to adjust TGP’s level of effort as needed to accommodate requests that take more than the normal effort. Supplemental Management Recommendations Three additional management recommendations were considered of such significance that they received supplementary discussion in the body of the report: (1) Reassign management functions For the TGP design, the Team recommends that the each of the major management functions identified below be clearly put forward as an option for one of the following entities to manage: the SO/RP team itself, its TA Contractor, the TGP Contractor or the Egyptian Partner Institution: • Needs assessment and development of a medium-term Training Plan • Annual plans and budgets • Training program design (course specifications) • Procurement • Processing of overseas training • Monitoring, follow-on and evaluation With the funds being managed at the SO/RP level, as the Team recommends, the training man￾agement functions can flow to the most competent organization to manage. The premise to be guarded in the future is the flexibility for the SO/RP team to decide the best responsible author￾ity for various training management tasks, within certain parameters (for example, that long-term participants be handled by the TGP Contractor). (2) Develop advisory services The Team recommends that IIE begin now to shift from a directive strategy to an advisory strat￾egy. The current strategy intended to ensure quality training based on Best Practices rests on the premise that if each SO/RP team submits to a set of standardized forms and procedures (needs assessment, plan, specifications, evaluation, etc.), then training programs will be of uniformly high quality and lead to Level 3 or 4 impact in their institutions. Yet the DT2 implementation experience has shown that imposing administrative procedures, albeit well-intentioned, did not result in better training. Later modifications resolved that problem by loosening procedural re￾quirements, but IIE now has less input into decisions about training that affect its quality and usefulness. It should start now to move toward the full-fledged advisory service capability that should be a part of the TGP design. (3) Build the organizational capacity of local training organizations Evaluation and Options Development for DT2 Page xv USAID/Egypt Aguirre International The TGP Contractor should provide services to local training providers that help them develop their organizational capacity and readiness to address the training needs of USAID and other lo￾cal organizations. Sustainability should be built into the TGP design so that local organizations increasingly assume the training management responsibilities currently handled solely by U.S. contractors. Core funds should be provided in TGP to promote local institution strengthening. As a result, USAID training done in Egypt should improve in quality, as local training organiza￾tions improve their practices, and it should become sustainable, once USAID reduces its program in Egypt. As a result of these changes, training should become more integrated with technical assistance, as TA Contractors take more responsibility for planning these two kinds of support in tandem. USAID's training system should become more informed by Best Practices, as advisory services replace mandated, intricate administrative procedures. And local capacity to manage training should grow, as TGP turns some of its attention to assisting training organizations. Page xvi Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Evaluation and Options Development for DT2 Page 1 USAID/Egypt Aguirre International I. Purpose and Methodology In 1994 USAID/Egypt began the design of an innovative training management system that was intended to accomplish multiple objectives that if achieved, would resolve several major issues that had been raised at the Mission. The Development Training 2 ("DT2") project design was subsequently approved in 1995 and a contract let in 1997 with the Institute for International Education (IIE) to manage all participant training and a signifi￾cant portion of the in-country training. In the fall of 1998, after two of the six years of im￾plementation of the DT2 project, an evaluation was conducted that highlighted a number of management problems that were raising concerns, both with Stra￾tegic Objective and Results Package (SO/RP) team members and Technical Assistance (TA) Contractor staff who were interacting with IIE. The findings and recommendations of that evaluation provoked major changes in several key aspects of IIE's imple￾mentation. This evaluation has been conducted two years later after four full years of implementation to re￾view the changes undertaken from January 1999 onward in light of the needs of USAID/Egypt in managing training and to put forward options for the Mission to consider after the DT2 project ends in 2003. A. Background The design of DT2 flowed from several sources beginning in 1992. A consultant report in the fall of 1992 noted the following problems with regard to the implementation of USAID's large training portfolio: • Lack of management oversight due to existing management burden within the technical of￾fices; • Lack of Mission interpretation of the requirements of Handbook 10 regarding pre-departure orientation, follow-up and evaluation of participant training; • Decentralization of the management of participant training meaning that approximately 30 technical contractors were implementing participant training; "DT3" ? Instead of referring to the follow-on to DT2 as DT3, this report uses a new ac￾tivity name proposed by the Team: Training for Global Performance "TGP" Throughout the report TGP will therefore refer to the mechanism through which USAID/Egypt will manage its training from 2003 onward. This new activity name is a proposal only and was not the subject of extended discussions with USAID staff. Page 2 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt • Lack of TA Contractor capability and experience to implement participant training following the requirements; • Lack of consistent use of orientation, follow-up and evaluation by the TA Contractors. The situation was made even more confusing by the practice whereby TA Contractors used their sub-contractors to implement participant training. In this environment it was impossible for the USAID technical offices to participate in, oversee or assess the quality of the training programs managed by contractors or to measure any results from their investments in participant training. Internal control assessments in 1991 and 1993 and an audit by the Office of Regional Inspector General in 1994 were conducted that highlighted the difficulties of ensuring compliance with Handbook 10 requirements with over 30 contractors handling participant training. The 1994 audit concluded that the existing problems would be resolved once the Mission shifted to a cen￾tralized participant training contractor, as had been proposed at the time. A November 1993 memo authored by the DT2 principal designer, Diane Leach, articulated three options available to the Mission to correct the situation (maintain the status quo, require TA Contractors to improve, or centralize the funding and management). The Mission chose the last option primarily due to concern that the TA Contractors were, as stated in the Action Memo, "deficient in the OFE [orientation, follow-on and evaluation] areas" and that "just as we should not ask a training specialist to design and build a wastewater system, why should we expect … civil engineers to be training specialists." The original justification for centralized training management rested on rectifying the provision of OFE in USAID/Egypt's participant training program. However, the eventual Project Paper (issued on August 20, 1995) included tasks and responsibilities considerably beyond ensuring that OFE were properly conducted. For instance, since Best Practices for "results-oriented" training were being developed in other areas of USAID, their principles and techniques were in￾corporated into DT2. Likewise, reengineering was getting into full swing and influenced the DT2 design in significant ways. For example, the Project Paper stipulated that DT2 would col￾laborate closely with Mission SO/RP teams in assessing the training needs of partner institutions well in advance of the pre-departure orientation originally expected from the central training management contractor. The DT2 design therefore grew to embrace modern training principles and approaches in addi￾tion to mandating that "Best Practices" be applied to all Mission-funded participant training, in￾cluding "non-project" training.* To reinforce the message, DT2 was also charged with managing selected in-country training programs where contractors might not have a comparative advan￾tage. * A condensed version of USAID's 9-part series entitled Best Practices for Results-Oriented Training is included in the Annexes and includes definitions and examples of Best Practices that can be used by Mission SO teams. Evaluation and Options Development for DT2 Page 3 USAID/Egypt Aguirre International The DT2 project therefore aimed to fulfill the following objectives: • Correct non-compliance in implementation of overseas participant training (principally caused by lack of OFE, inconsistent application of Handbook 10, etc.) • Streamline training management (centralize funding to ease management burden, dismantle the large USAID/Egypt Training Office) • Ensure that training contributed to Mission SO/RP results (encourage Best Practices and needs assessments, selection by organizational need, training design linked to desired change, workplace follow-up, etc.) All of the attention in the early design years was on centralized management, not centralized funding. Funding was expected to be from various technical offices through PIO/Ps (the docu￾ment used to earmark and commit funds for participant training) and from several large central training programs for PIO/Ts (to fund the core management services). The idea of centralized funding apparently surfaced as a convenience and was not a feature in the original design articu￾lated in internal memoranda. A critical assumption upon which DT2's success rested, in terms of its ability to introduce Best Practices and bring about consistency in the implementation of OFE, was the extent to which the USAID Mission became reengineered. The signifi￾cance of this assumption (see box) cannot be overemphasized in light of the disappointments DT2 encountered in its first two years of implementation. B. Purpose of the Report The Scope of Work for this report is entitled "Evaluation and Options Development" and called for two independent teams to consider the three components of the DT2 project. The first team evaluated the Integrated English Language Program II (IELP-II) and the English Language Testing and Training Program (ELTT), both of which were funded under the DT2 umbrella. That report is not a part of this document although its findings and recommendations were made available to the Team. The second team evaluated the training management component of DT2, the subject of this re￾port. Although the three components of DT2 are linked in that the IELP-II activity is the single largest user of DT2 services, especially regarding participant training, and the ELTT activity or￾ganizes English language testing and training for participants and in-country trainees, in reality the management of in-country and participant training is largely independent from the other two. Key Assumption The level of USAID reengineering would be sufficient to enable the SO/RP teams to determine train￾ing needs in collaboration with their TA and Egyptian partners. Page 4 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Staff at USAID/Egypt, TA Contractors, Egyptian partners and IIE all refer to the management of training as "DT2" even though technically the IELP-II and ELTT components are included in the DT2 funding. This report also adopts this terminology and refers to the IIE-managed training component as "DT2." References to the other two components of DT2 will indicate "IELP-II" or "ELTT." This report considers only on the "DT2" component and leaves to the first team the responsibility of proposing findings and recommendations regarding IELP-II and ELTT. The purpose of this report as put forward in the Scope of Work was to… Develop options, based on an evaluation of DT2, for the management of train￾ing, participant and in-country, after the end of DT2 in 2003 and conduct a full￾mission Training Options Meeting to consider evaluation findings, options on training management issues, and strategies to improve Strategic Objective Teams' participatory role in integrating training into results package activities. Below are extracts taken directly from the SOW of the principal questions that the evaluation was to address (the Scope of Work and the Team's Work Plan are included in the Annexes): Centralization of training. To what extent has centralization helped promote best practices within SO teams and their technical contractors? What have been the gains and losses resulting from the centralization of participant training? Has the centralized approach resulted in more consciously integrating training with other interventions? DT2-specific approaches How well were Training Needs Assessments done? How effective is training based on TNAs vs programs not based on them? To what extent did the level of success of DT2 among various SO teams depend on their level of re-engineering? Can any conclusions be made regarding the impact of DT2's approach to training, the training delivered and its impact on target organizations and individuals, on SO results? Funding. What was the effect of the high level of funding and the way it was allocated on the working relationship between IIE and SO teams and technical assistance contractors? What are the pros and cons of allocating funds directly to DT2 vs. via SO teams? In-Country Training. Who should manage generic in-country training? Technical? What criteria should be used by SO teams in deciding whether to incorporate an in-country training program in a technical assistance contract or have it managed by a central contractor? IIE Management. What role did the level of rigor on the part of IIE play in the receptivity of SO Teams to improved training practices? What were the effects of IIE's more recent flexibility following the 1998 evaluation? What IIE management innovations have worked? How well is the centralization of TraiNet under DT2 working? Evaluation and Options Development for DT2 Page 5 USAID/Egypt Aguirre International Overall Impact. How well is the Mission's training portfolio serving women and geographic ar￾eas targeted in the new Mission strategy? How well is it serving Egyptian private sector needs? What features should a new model have to promote new Mission strategic needs? How can the Mission best promote stronger integration of training? What options other than a centralized model are there to promote this? This report provides insight to these questions and proposes options for the Mission to consider in planning for the management and funding of its large training portfolio from 2003 onward. C. Methodology and Survey Population The Team established that there were four key products that should result from the "evaluation" for USAID/Egypt: • An analysis of the features of DT2 that have been effective and should be continued • An analysis of the features of DT2 that have been problematic and should either be discon￾tinued or modified • A review of options that USAID/Egypt should consider for "post-DT2" training • Recommendations for which options would best serve the needs of USAID/Egypt. The Team then eliminated the following two approaches that might be traditionally associated with an evaluation but would not produce the type of data needed: • A statistically-significant survey of returned participants to determine the level of impact of training on their work units was beyond the Scope and not the primary focus of this "for￾ward-looking" evaluation; and • An evaluation of contractor performance was not the primary focus of the Scope. In view of the forward-looking nature of this evaluation, the Team decided to downplay the use of the term "evaluation" since the above methods were not to be used. With thousands of Egyp￾tian participants sponsored by DT2 since 1997 from scores of institutions, no weighted random sampling of significance could be elaborated and administered that would divulge any useful in￾formation on the impact DT2 training might have had. Moreover, no baseline could be deter￾mined at those institutions against which an evaluator might measure impact or organizational performance change. Although this report is not an evaluation of contractor performance in implementing DT2, the Team nevertheless needed to interview the users of DT2 services as provided by IIE to deter￾mine the perceived level of quality and identify areas for improvement. Questions probing into the DT2 customer satisfaction level were therefore included in the Team's methodology. Page 6 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Having eliminated two traditional methods employed by evaluators, the Team adopted the fol￾lowing guidelines: • Structured interviews would be the principal tool for Team members to use in gathering in￾formation. • Standardized as well as opened-ended questions would be included in the interviews to en￾able the Team to quantify selected responses to supplement the more “anecdotal" informa￾tion gathered • The major users of DT2 would be interviewed at USAID-Egypt (SO/RP, Program and pro￾curement staff), TA Contractors and partner institutions. Approximately half of those inter￾viewed would be involved in the activities selected as Case Studies. The other half would have had some significant relationship with the DT2 Project. • Case Studies would be developed to reveal the dynamics of training management in Egypt as seen from within a single completed training event. The case studies would be selected ac￾cording to criteria the Team had established and would involve multiple training "programs"* The Mission provided a list of contacts that covered all of the USAID/Egypt staff directly and indirectly involved in DT2. Because the list of TA Contractors produced was so large, the Team employed a Case Study approach to cover a sampling of the TA Contractors. In this way the Team could cover in more depth a small number of TA Contractors to obtain an inside view on the management of a range of training issues, from needs assessment to follow-on. Such a holis￾tic approach had the advantage of possibly uncovering examples of management accomplish￾ments and problems as well as creative ideas that could assist the Team in developing workable options for the Mission to consider. (A complete list of individuals interviewed is included in the Annexes.) An Interview Questionnaire was designed so that each of the Team's four members would gather information in a consistent way that could be easily compared. To the extent possible, the inter￾view guide included quantitative questions in order to be better able to compare data objectively. Finally, a number of open-ended general questions were asked in order to stimulate fresh ideas for the Team to consider for the future management of training. The interviews lasted between 45 and 60 minutes with the questionnaires being completed by the Team members rather than the respondents. At the end of the third week, Team members organized data from all the Question￾naires in order to tabulate responses and bring out the range of substantive comments from open￾ended questions. * An extensive collection of documents relating to DT2 were assembled and studied by the Team. * Information on the USAID-funded "projects" and activities chosen as Case Studies -- such as the name, a brief description and the criteria used for selection -- is included in the Annexes. * A copy of the Interview Questionnaire and the Powerpoint presentation are included in the Annexes. Evaluation and Options Development for DT2 Page 7 USAID/Egypt Aguirre International The Survey Population Ninety-one structured interviews were conducted by the four-member team over a period of two weeks. Eight Case Studies were developed, four considered "primary" and four "secondary" by the Team. All key staff at IIE/Egypt were interviewed, as well as senior managers at IIE's home office. The Team also interviewed by telephone the principal designer of DT2, the first IIE Chief of Party for DT2 and the software specialist and USAID contractor responsible for TraiNet. Because a random, statistically significant sampling could not be used, for the reasons discussed above, the quantitative findings cannot be considered "representative." However, such a large number of users (83) of DT2 services were in fact interviewed by the Team that the qualitative information produced carried unusually high weight. Through these in-depth interviews, the Team believed it had uncovered most of the perceptions, problems and new ideas that one could gather regarding the way training had been managed under DT2 and the options for the future. The Case Study approach was particularly useful in that the Team could assess multiple aspects of training management from within a single project or contract. In several instances, Team members attended training programs and were able to interview Egyptian institu￾tional partners and participants. The numbers and affiliations of the survey population are shown in the table. The num￾ber of Training Providers interviewed was small due to the emphasis the Team placed on the Case Studies and meeting with SO/RP members. Although the Team had targeted interviewing approximately 50 people, in fact they were able to complete Interview Questionnaires on 83 in￾dividuals from the categories shown. This sampling breadth added significant value to the find￾ings. D. Organization of the Report The report is organized as follows: Chapter I establishes the context for assessing DT2 at this point in its implementation and the objectives for this report and the methodology employed. Chapter II breaks out the processes used in DT2. Each section assesses the role played by the process or activity in question in managing training using Best Practices and explains the find￾ings of the Team based on the data gathered and analyzed. Chapter III presents the Balance Sheet on DT2's accomplishments, puts forward the key recommendations drawn from the analy￾sis in Chapter II and synthesizes all the findings and recommendations into several Options for USAID/Egypt to consider in a "new DT2." Throughout the report, the term "DT2" refers to the present USAID/Egypt activity begun in 1997 that centralized the management of Mission-funded participant training, and a significant portion Organization Number USAID/Egypt 32 TA Contractors & EPIs 38 Training Providers 4 Participants 9 Total 83 Page 8 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt of in-country training, under a single funding source. The activity is managed by the DT2 Con￾tractor, the Institute for International Education (IIE), selected in 1997 through competitive pro￾curement. A distinction is made in this report between DT2 and IIE. When referring to the DT2 design features, funding mechanism or overall program, the term "DT2" is used; when referring to spe￾cifics about contractor management of the DT2 activity, or the contractor's procedures, perform￾ance or approach, the term "IIE" is used. When referring to the future design of Mission-funded training, the term "TGP" is used, for Training for Global Performance, the name proposed in this report for what has been called "DT3." Evaluation and Options Development for DT2 Page 9 USAID/Egypt Aguirre International II. Findings A. Principal Issues Each section that follows includes a description of the issue, a list of the Team's findings and a discussion. 1. Assessing training needs The process of analyzing performance gaps and designing solutions has been thoroughly elabo￾rated, tested and applied for many years in North America, in particular by the private sector. Extensive information is available on various forms of performance gap analyses which will not be repeated in this report.* Conducting TNAs carries the obligation that a proper performance analysis has been conducted and that an intervention involving training is appropriate. For too long USAID-funded training world-wide has been sponsored that may have led to increased knowledge and skills on the part of individuals but did not necessarily result in any institutional performance change. Part of the reason is a reliance on documenting training requests rather than on an analysis of performance needs. If Best Practices are applied faithfully, the process of assessing training needs follows the per￾formance gap analysis. With DT2, the basic assumption was that the SO/RP team, usually through its TA Contractor, would first identify the organizational performance changes in the SO's partner institutions that if made, would bring about measurable improvements that were linked to the Intermediate Results stated in the Strategic Objective. The process, referred to in the DT2 design and by IIE as a "Training Needs Assessment" (TNA), was to address the fol￾lowing questions: • What are the performance gaps in the partner organization that are hindering progress to￾wards achievement of the desired results? • Which, if any, of these performance gaps can be filled by training? • Which individuals, or positions, are key to producing the desired changes needed to close the performance gap? • What knowledge, skills and attitudes (KSA) do these individuals need to learn in order to make the changes? The use of the term "TNA" introduced some confusion in the process that might, or might not, have included "performance analysis." In some cases an analysis of performance gaps at an or- * The Annexes contain a Quick Reference Guide to Best Practices in which these terms are discussed and illustrated. Page 10 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt ganizational level did occur within the parameters of a TNA, especially in the early years,; but in many cases the TNAs were more limited to identifying "training needs." Following the needs assessment, training may be proposed as a solution to some or all of per￾formance gaps identified. Training is but one type of intervention and may not lead to improved performance. If the performance gap analysis suggests that the accounting department should move from manually-completed ledgers to a computerized financial system to meet future de￾mand, no amount of staff training in new software will close that gap if computer equipment is not procured, installed, adapted to the environment and made operational prior to training. Technical assistance and training have their greatest impact on organizational change when planned and implemented together. In some respects each contains major elements of the other: TA is ideally the transfer of knowledge and know-how from "experts" to local counterparts, ac￾complished through training (on-the-job, counterpart, workshops, etc.). Training ideally in￾cludes technical assistance so that the KSA acquired is sustainable and applicable at the work place. An effective training program, for example, would include visits by the trainers them￾selves to the organization's work unit, preferably before and after the training, where they would assist the participants (through "TA") apply their newly-acquired KSA to bring about organiza￾tional performance improvements. Training with TA, and TA with training, are powerful tools for economic development, when properly integrated and designed. There are many kinds of effective TNA’s: formal or informal, lengthy or brief, in someone’s head or on paper. Two key characteristics of a useful TNA are that the assessment take no longer than necessary to provide the essential information and that the resulting report be “user￾friendly”. That is, the information the TNA provides should be as brief as possible and it should provide the answers to the questions listed above, in a manner specific enough to enable the user to develop clear training specifications and select appropriate participants. A TNA is time sensi￾tive – as organizational needs evolve, the TNAs should be supple enough to remain useful and to be updated easily. Findings • Understanding that the process of assessing training needs was an integral part of reengineering and part and parcel of Best Practices, IIE sought to apply the process with vigor from the beginning as called for in its contract and in the DT2 design. • The TNA models used by IIE in the early period were too lengthy, costly and in some in￾stances, were isolated from stakeholder input • DT2 resources expended to complete thorough TNAs in the first two years were not in pro￾portion with the anticipated training volume • Training programs were delayed as a result of the TNA "requirement" and resulted in signifi￾cant USAID training pipeline build-up Evaluation and Options Development for DT2 Page 11 USAID/Egypt Aguirre International • Early TNAs were inconsistent in quality and often did not lead easily or directly to training design • Needs assessments based on generally available sector assessments may not meet the needs of specific local institutions. • Some early TNAs were sector surveys rather than organizational needs assessments • IIE data suggest no association between a lengthy and costly TNA and the results obtained as indicated in subsequent IIE on-site evaluations • IIE responded to the lack of support for the type of TNAs conducted in the early years by eliminating this requirement, reallocating resources to implementation (to unblock the pipe￾line) and considering new approaches to assessing organizational needs • IIE currently either implements training requests without requiring a needs assessment or uses a variety of formats, some of which are not written • The pendulum has swung from an overly lengthy process and product to a variety of briefer, more informal types of TNA; these are also of inconsistent quality, and many are not avail￾able in written form, even those undertaken with care, incorporating “Best Practices”. • New ways to analyze performance gaps, such as using rapid appraisal techniques, have not been tried. • The absence of a needs assessment has allowed in some cases uncritical training or training that was not the solution to the problem • In general, TA Contractors now take the lead in performing TNAs and generating training requests. Although information is lacking as to how these training plans and TNAs are done, data suggest that many training programs from TA Contractors are linked to RP results sought. Discussion When IIE began implementation of the DT2 contact in 1997, reengineering was new at USAID/Egypt. The SO/RP teams were just being formed and undertaking TNAs fell relatively low on any list of priorities. With its clear mandate to arrange for what were labeled in the IIE contract as "micro" level TNAs, the contractor bore the de facto burden of carrying out the TNA process. Although the intent was to produce high quality products that exemplified “Best Prac- Page 12 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt tices”, the process was so slow that pipeline problems developed, and the numbers of trainees were reduced significantly. A general survey of a sector, although called a TNA, would not likely yield the specific informa￾tion needed to plan courses and select participants. On the other hand, an in-depth "micro" in￾vestigation of an organizational unit, however detailed, did not necessarily lead to a training de￾sign either, becoming instead a position-by-position inventory of skill needs that might be based more on the employee's training "needs" than linked to an organizational performance target. The common error of confusing a legitimate TNA with a "training wish-list" characterized some of the TNAs reviewed and added no measurable value to the Best Practices process of ensuring that training is linked to organizational change. Some TNAs reviewed were nonetheless useful. For example, the 24-page TNA prepared in August 1997 for the Egypt Utilities Management Project (EUM) under SO6 found that so many changes were needed in the relevant organizations that it would not be productive to recommend training for the staff until certain organizational conditions were changed. Accordingly, the TNA recommended a “top-down” approach to training in which upper-level management offi￾cials of the water utilities would be trained first. Subsequent TNAs were modified and training was provided as the work culture began to change. Approximately $1.3 million was spent on TNAs in the first two years of DT2. By itself, this amount cannot be considered high in that a well-executed performance analysis can and often does lead to more effective training and results. If USAID/Egypt were to spend even two per￾cent of $100 million, or $2 million, to analyze and plan for training correctly, that investment could yield tremendous returns. The savings alone from training not funded because it was ex￾traneous to the needs identified in a proper TNA would be worth the investment. Unfortunately, few of the TNAs reviewed led to improved training or added significant value to the process of designing and planning training that would lead to organizational change. Instead, they were more often than not conducted by outside consultants, were far too large and even ir￾ritated the stakeholders (especially staff from the SO/RP teams and TA Contractors). To verify the generally negative anecdotal information collected on the TNAs, the Team asked IIE to cross-tabulate its evaluation data with the size and cost of the TNAs. For instance, train￾ing programs that were evaluated by IIE as having achieved Level 3 results ("application of KSA acquired) were compared with the TNAs that formed the basis for those training plans. Not only was no positive association found between large, comprehensive TNAs and training programs considered to have produced results ("successful"), but a small inverse relationship appeared where successful training programs fared better with small or no TNAs! The Team concluded that the TNAs introduced and completed by IIE in the first two years of DT2 did not produce the desired results. The conclusion to be drawn concerning the lack of value added was that the type and scale of the TNAs, and their lack of stakeholder involvement, did not fit the need. Evaluation and Options Development for DT2 Page 13 USAID/Egypt Aguirre International Managers at IIE responded to the criticisms in the January 1999 evaluation with marked changes in the way it implemented the contract-mandated TNAs. Since then several types of TNA have been employed, examples of which are included in the Annexes under "Training Needs Assess￾ments". 2. Planning and processing training The best performance analysis followed by a faulty training design and specifications produces disappointments. The expectation in DT2 was that training needs assessments would form the basis for a multi-year Training Plan. How has DT2 fared with regard to the all-important process of developing training plans and effectively processing participants? a. Training Design & Specifications Each of the approximately 40 SO/RP teams at USAID/Egypt is asked to approve a plan specifi￾cally for training funded and managed by DT2. The purposes of the Training Plan are to help SO/RP teams, including their Egyptian institutional partners, think through their medium-term needs for training and to help DT2 managers budget funds and IIE staff resources to manage this training. The Training Plan currently used is a matrix of annual training activities that provides information on training venue, its frequency, the number of trainees per event, its duration, esti￾mated start-date, and other similar information. The SO/RP team leader and the DT2 CTO ap￾prove each plan. The TA Contractor or SO/RP team member may also request ad hoc training programs that are not part of the overall Training Plan. Each ad hoc activity requires these same approvals. Based on an approved plan, the TA Contractor uses the IIE Training Specifications Form to de￾scribe in detail each training activity. IIE asks to receive the completed "specs" two months in advance of the start-up date for off-the-shelf short programs, four months for tailored programs and eight months for long-term overseas academic programs. * Findings • A three-year training plan, and even an annual plan, containing sufficient detail, lacked the flexibility that some SO/RP teams required to address actual training needs. The process of approving ad hoc training programs is seen by some as labor intensive and difficult to achieve in the time available. On the other hand, ad hoc requests obviate the need for more time-consuming TNAs are can be viewed as labor-saving. A clearly agreement on the type an depth of information needed to back up training requests is needed. * The specifications form and information about this process are described in more detail in IIE's DT2 Users' Guide to the DT2 Project. Page 14 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt • TA Contractors sometimes find it difficult to answer the questions and provide the informa￾tion required in the Training Specifications Form. • IIE has provided assistance in developing training plans and specifications for a wide variety of training activities. In many instances the SO/RP teams and TA Contractors have been satisfied with IIE's services. • Despite the significant improvements made by IIE in articulating training plans and program specifications over the last two years, more needs to be done to simplify the process. In some instances forms are still being completed that have little purpose or whose use is not critical to the smooth functioning of the training process. The degree to which the training management is uncluttered affects the level of customer satisfaction with IIE's performance. Discussion In some cases TNAs were updated or modified in order to ease the way toward a clear training design (training specs). There was some confusion over what should be included in a Training Plan. What is its purpose and scope? Should it be a list of training activities with dates and subjects? Should it include the names of people to be trained? Is it a budgetary document only, or a guide to selecting a training provider? The Training Specifications Form asks for information requiring both familiarity and experience with institutional performance analysis, such as • Current work performance in relation to desired performance • Non-training factors that may affect achievement of results • Expected result of training, in terms of specific job performance • How results can be measured • Specific training objectives (knowledge, skills, attitudes to acquire). The relevance and usefulness of the training program depends to a large extent on the accuracy of this information, yet in some cases TA Contractors have difficulty in thinking it through. While in some cases, IIE staff and the TA Contractors they serve have a positive working rela￾tionship that allows them to communicate this information, in other cases; the TA Contractor feels that the process is unnecessarily demanding. b. Procurement At present, the bulk of IIE's work is procuring training services that can be categorized into five types: • In-country, off-the-shelf programs • In-country tailored programs Evaluation and Options Development for DT2 Page 15 USAID/Egypt Aguirre International • Overseas, off-the-shelf programs (including the "customized" observation and study tours) • Overseas, tailored programs • Overseas (U.S.) long-term academic programs. The USAID jargon considers that trainees attend in-country training whereas participants attend overseas (that is, U.S. and Third-Country) programs. Procurement of most of these programs entails a contractual arrangement between IIE and a training provider, be it a for-profit training firm, U.S. university or an NGO. The Observation and Study Tours (OSTs) to the United States are managed by IIE's Washington office in most cases and those to third countries can be man￾aged by an IIE office in that country or with the help of a TA Contractor or host-country gov￾ernment or institution. The management requirements vary considerably according to the training type. For off-the￾shelf programs in Egypt for groups, such as in financial management, IIE will issue a Request for Quotations (RFQ) and qualify organizations that submit proposals that meet pre-defined stan￾dards. Depending on the content and location of the training, both Egyptian and U.S. organiza￾tions with Egypt offices may submit proposals to be considered for the training. Some in￾country training is considered "technical" but not "tailored," such as a course in proprietary soft￾ware certification that IIE will procure for a group of Egyptian trainees. In many cases IIE pre￾qualifies a number of Egyptian organizations to streamline and expedite the procurement proc￾ess. For U.S.-based off-the-shelf training, IIE's Washington Office will identify the appropriate courses or follow IIE/Egypt's instructions in placing participants in training. For U.S.-based tai￾lored programs, IIE has developed lists of organizations likely to be interested in bidding and will issue an "RFP" by email. TA Contractors in Egypt may recommend organizations to receive the RFP – there is no short-list or pre-approved list of training providers for U.S.-based training. After IIE advertises its RFP, it constitutes a Technical Review committee comprised of IIE staff and a representative from the TA Contractor, if applicable, familiar with the Egyptian stakeholders. A training provider is selected on the basis of technical standards and cost. Complex tailored programs require intensive management and negotiations with other parties, particularly the TA Contractor and the Egyptian partner institution. DT2 has been encouraging in-country off-the-shelf training, as it is much less expensive, generally of good quality, and more efficient to procure. To date, about 72 percent of the training programs have been in Egypt. The portion of these programs that are off-the-shelf grew from 36 percent in 1999 to 79 percent in 2000, as DT2 encouraged TA Contractors to use the growing number of programs publicly available in Egypt. Overseas academic programs also require intensive management, including the processing of vi￾sas and placement. Although less than one percent of the participants attending training pro￾grams funded by DT2 have been in long-term academic programs, the workload necessary to Page 16 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt prepare, test, place, monitor and re-integrate long-term participants is far greater than one percent of IIE's current Level of Effort. Findings • Most users are now satisfied with DT2 services and refer to problems in the past tense. The serious problems highlighted in the 1999 evaluation have been largely resolved. For the most part, DT2 has handled procurement with no significant problems, removing an enormous workload from the Mission's former Training Office. • In-country procurement of courses has produced the unexpected advantage of strengthening the planning and management capacity of some local training firms. • Though in most cases USAID/Egypt and its TA Contractors have been satisfied with IIE's procurement of short training programs, difficulties have emerged in the procurement of tai￾lored training programs for the private sector. Tailored training is more management￾intensive and often requires negotiations between the TA Contractor and the training pro￾vider. Although IIE does have certain technical expertise on its staff to advise on the appro￾priateness or quality of training content, with the exception of training in highly technical subjects, it has occasionally been caught in the middle of these negotiations. • On several occasions, USAID/Egypt has asked IIE to support the design of large training programs and even to administer them, as a kind of surrogate "TA Contractor." These ac￾tivities have diverted IIE staff resources for project management tasks that were not factored into the IIE contract level of effort. Discussion The difficulties managing training programs for the private sector stem, in part, from the need to respond quickly to perceived demand. The approval procedures of DT2 have slowed down this process when one of the two signatories required at USAID were not immediately available for approval or when the TA Contractor does not supply timely, adequate training specifications. Except for approving or disapproving, neither DT2 nor the SO/RP teams have had noticeable involvement in participant selection. These decisions are generally left to the TA Contractor or the partner institution. While there are no data that show the quality of participant selection in terms of their potential to apply the fruits of their training to their work place, there is a widely￾held view that by better aligning training with SO/RP strategies, the number of inappropriately selected individuals has been reduced. The most notable impediment to the final selection of appropriate candidates is English-language testing. Many who would benefit from training in the United States do not pass the test, and of- Evaluation and Options Development for DT2 Page 17 USAID/Egypt Aguirre International ten, it is thought, trainees are nominated on the basis of their English language ability, rather than on the level of professional competence. * c. Processing overseas participants With assistance from the Washington and New York offices, IIE/Egypt places participants in the appropriate training programs in the United States. These programs include short courses, some internships, and workshops. Observation and Study Tours to the United States and, increasingly, to Europe, Asia, the Gulf states and sub-Saharan Africa, are arranged directly by IIE/Egypt. For all overseas participants IIE manages the process of arranging for English proficiency verifica￾tion, obtaining visas, ensuring medical authorizations, purchasing airline tickets and providing travel allowance advances. Processing an overseas training participant can require up to roughly 25 documents, many completed by different parties. The process is in part intended to ensure ad￾herence to USAID travel and immigration regulations, cost containment policies and participant eligibility for academic and other training programs. Following Best Practices and contract requirements, IIE must arrange for each participant to at￾tend an orientation intended to "maximize participants' readiness for learning, help them develop an action plan for applying their training to the workplace, provide cultural information, and sort out logistical and administrative issues" (DT2 Users' Guide). Findings • Generally efficient and satisfactory services are provided by IIE, in contrast to the pre-DT2 years when no systematic and comprehensive pre-departure orientations were being con￾ducted by the TA Contractors • Problems arise principally due to "special" cases – private-sector participants travelling on partially-sponsored travel with little time, clear objectives and sufficient means; or high￾ranking civil servants • The medical exam requirement remains challenging to administer due to cultural impedi￾ments in Egypt • IIE and USAID/Egypt have collaborated closely in trying to make the application of Mission regulations more flexible and efficient • The DT2 User's Guide has helped considerably clear the air of misunderstandings concerning USAID regulations and procedures * The problem of English-language testing has been addressed by the English Language Testing and Training (ELTT) evaluation team. Page 18 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Discussion With some exceptions, DT2 has processed overseas participants efficiently and to the satisfaction of all parties. Staff at USAID/Egypt expressed dissatisfaction at the previous high turnover of IIE staff that complicates the already complex procedures to process and prepare all participants for overseas training. Likewise some TA Contractors complained about poor communication be￾tween IIE and other parties about processing requirements. In some cases, friction between IIE and TA Contractors working with the private sector results from the different needs and interests of private sector participants. In contrast to public sector employees, business people often wish to travel on their own visas and airline tickets, so they perceive the value of the overseas training differently than civil servants who have few opportunities for international travel. Private sector participants choose to accept USAID programs because they see opportunities abroad for learn￾ing about markets, products, and production techniques. Yet they find it difficult to understand USAID's travel requirements, such as the medical exam, the English language tests, the required administrative orientations and the visa rules that prevent extending their stay in the United States for private business after the USAID-funded training ends. Training in third countries raises special concerns due to the USAID requirement that partici￾pants visiting non-code 941 countries obtain a waiver, which at the moment must be signed by no fewer than eight USAID officials, including the Director and Deputy Director. This require￾ment is particularly cumbersome for participants working on USAID-funded activities that target precisely those countries, such as businesses trying to increase export markets to Europe. It is counterproductive for USAID/Egypt to fund activities and then be prevented from using training that can help achieve those objectives. IIE and the CTO and other USAID/Egypt staff have been working together over the past ten months to streamline the Mission Training Order so that it conforms to ADS 253 guidelines without adding further prescriptions. 3. Monitoring, follow-on and evaluation Monitoring, follow-on and evaluation are all essential components of effective training. Below are definitions of these terms as employed in this document: • Monitoring is the process of assessing progress during training. • Follow-on is the process of providing additional support to a returned trainee, either to pro￾mote the continued acquisition of knowledge and skills, or to assist in their application at the workplace. • Evaluation implies that a judgment will be made about the quality of the training delivered and/or the results the training might have 3, 6 or 9 months after the training has ended. Evaluation and Options Development for DT2 Page 19 USAID/Egypt Aguirre International The Monitoring and Evaluation system used by IIE, is based on the 4 levels of Donald Kirkpa￾trick’s model, as outlined in USAID’s “Best Practices” document: Satisfaction Are trainees satisfied with the quality of training? Learning Did trainees learn or acquire the desired KSA? Application Did trainees apply new KSA in their organizational units? Results Did the KSA applied by trainees enable the organization to improve performance? Monitoring. In the case of in-country training, an IIE staff member attends the first session of all training activities. For short-term in-country training, monitoring includes a mid-course visit. In the case of longer-term training, IIE staff make periodic visits during the training. For long-term overseas training, U.S.-based IIE staff send periodic monitoring reports to IIE/Egypt. In some instances special requests by TA Contractors have supported direct communication between the TA Contractor and the U.S. training provider. Follow-On. All overseas participants are invited to participate in IIE's Follow-on Program. This feature offers a maximum grant of $1,000 for the following: the opportunity to purchase books and professional materials, or tuition support for workshops or courses. In addition, participants who wish to organize a course or series of workshops for others may submit a proposal with a budget and receive funding for such activities, separate from the $1000. There are also generic services available to alumni, including a special series of technical and general lectures as well as a newsletter produced by IIE. In-country trainees are ineligible for the Follow-On program. IIE staff explained that because the names of in-country trainees are not in their database, it is impossible to send them invitations to in-country events. The follow-on package noted above should be distinguished from follow-on visits that could be made to the work site by training specialists to assist returned participants in applying new skills. IIE currently provides no follow-on assistance of this type, although it did early-on. In some cases TA Contractors have contractual agreements to provide follow-on support in connection with training they manage. However, in many other cases TA Contractors provide no follow-on support whatsoever. The managers at IIE appreciate the crucial role follow-on plays in encour￾aging the application of KSA acquired during training and are considering ways to reincorporate workplace follow-on into their service package. Evaluation. DT2’s services include three levels of Evaluation: • Basic, covers Kirkpatrick Levels 1 and 2 (Satisfaction and Learning), the data for which is compiled from questionnaires completed by participants and trainees at the end of their training. All training includes this level of evaluation. For short-term participants, the Basic evaluation is the only appraisal done, aside from an oral debriefing upon return to Egypt. There is no on-site work place evaluation arranged by DT2. Page 20 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt • Partial is an intermediate level between basic and standard. This level includes the de￾velopment of a participant Action Plan to help evaluate Levels 3 and 4 (Application and Results). A single evaluation visit is made to the organization between 4 and 12 months after the training. The partial service is provided for certain kinds of short-term overseas training, and for tailored short-term in-country training. • Standard includes everything in the partial evaluation, but includes two or more post￾training evaluation visits to the work site. The standard evaluation is performed for all long-term trainees, whether overseas or in-country. When evaluation visits are made by IIE staff to assess Application or Results in the work place, they meet with an individual trainee or a group, and also make every effort to include the super￾visor and the TA Contractor. In some cases the TA Contractor arranges the meeting. In some cases the training provider has a contractual obligation to provide level 1 and 2 evaluations and must provide the verifiable learning objectives to be assessed for level 2 evaluations. The IIE evaluation specialist typically prepares the Monitoring and Evaluation Report following the site visit. In line with IIE's effort to streamline processes and reallocate resources to training implementation, M&E reports have become shorter and incorporate less narrative. Findings • IIE's monitoring of participants while they are in overseas programs works well and is occa￾sionally supplemented by a TA Contractor's contacts or systems • IIE's follow-on programs are innovative and involve seminars, workshops and a $1,000 grant for participant use to further learning; however, the administration of the participant grant program is not always well-understood by participants or TA Contractors • A number of interviewees expressed some confusion about the actual procedures for access￾ing the Follow-On funds. • IIE provides no follow-on services in the workplace, although in some cases the TA Con￾tractors may be providing such services • IIE's evaluation system has been significantly streamlined since January 1999 and resources diverted to more urgent needs, such as implementing training • It was unclear whether evaluation results were used to improve future training. Few DT2 customers could be found who value or make effective use of the evaluation reports, which are intended to provide information that could greatly enhance the application of KSA that lead to organizational performance improvements. Evaluation and Options Development for DT2 Page 21 USAID/Egypt Aguirre International • The overall process for Monitoring, Follow-On and Evaluation is flexible, allowing for variation in whether IIE or a stakeholder provides the services. Some of these variations are specified in DT2 contracts; in other cases it is unclear who is providing which services. Discussion IIE staff, in discussing the possibility of restoring the practice of follow-on visits in the work￾place after training, commented on the apparent “disconnect” between the job of the evaluation specialist and the follow-on that could be provided by a TA Contractor or by a IIE follow-on specialist. Some evaluation staff have noted that since they go in to assess application of KSA in the workplace, they might be the logical ones to continue with follow-on visits, or to collaborate with TA Contractors so that their evaluation findings could be incorporated into TA follow-on support in the work site. 4. Training data collection and management Data collection on participant and in-country training in Egypt is treated below in two categories, according to the user of the information. USAID/Washington and USAID/Egypt The regulations governing the collection of training data are contained in ADS-253 and have been made part of OMB requirements for USAID. Missions are mandated to collect data in the agency-authorized TraiNet software or, if not yet operational, its forerunner, PTMS (Participant Training Management System). TraiNet is operational in Egypt and is maintained by the DT2 Contractor. For many years USAID/Washington has tracked participant training information to use princi￾pally in reporting to Congress and others. The data also provides the agency's central bureau (G/HCD) with the basic information needed for analysis of training trends and costs. On September 22, 1999 the USAID/Egypt Director stated that both he and the U.S. Ambassador were "keenly interested in USAID training data" and encouraged SO/RP team leaders to assist DT2 in ensuring that both in-country and participant training data were collected. He noted that beginning in FY2000 "it will become a quarterly activity to collect and report data on all USAID-funded training." The data collected by TraiNet today builds on information gathered in the previous system. Par￾ticipant training is defined as that which occurs outside the country of the sponsoring Mission, as compared to in-country training. Page 22 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt • Participant Training: Participant name and country of origin, gender, sector of employment, field of study during training, language, training location, provider and duration, and direct cost of training. • In-Country Training: Subject area of training, total trainees per group (with gender break￾down), the total cost for training with the direct (unburdened) cost disaggregated. Only in-country training of more than 3 days (or 15 hours of contact training scheduled inter￾mittently) need be collected by USAID/Egypt according to ADS-253. No individual names need be entered into TraiNet for training in Egypt. Data Linked to a SO/RP Team or TA Contractor's Activities Although TraiNet is said to have the capacity to assist in the management and evaluation of training, it distinguishes itself from earlier PTMS versions of the software by including reengineered features that link the training objective to the SO team's Intermediate Result. This innovation was thought to aid in promoting Best Practices by forcing training managers to iden￾tify the link, or ensure that it existed, upstream to the SO team's achievement of its IRs. Findings • TraiNet collects participant training data of use to USAID/Washington and to a limited de￾gree, to USAID/Egypt • The data is collected by DT2 and some TA Contractors who comply with the USAID Direc￾tor's memorandum of 1999 • Participant data is believed to be accurate in view of the fact that all but a small number of overseas participants are processed by DT2/IIE • In-country data is believed to be incomplete because so much is implemented by TA Con￾tractors who do not submit the data to IIE; however, all in-country training managed by DT2 is recorded in TraiNet • Most contractors, including DT2's, have developed a customized data management system that duplicates and supplements TraiNet because no contractor was able to adapt TraiNet's software sufficiently to address their needs • TraiNet therefore provides little value added in terms of data tracking and reporting, either to the TA Contractors or to USAID Egypt • When IIE analyzes and reports data on its training portfolio, as it did for the Team, it prefers to use its proprietary data management system over TraiNet. Evaluation and Options Development for DT2 Page 23 USAID/Egypt Aguirre International • There is no evidence that any SO teams saw or used data produced by TraiNet. Discussion Interviews with TA Contractors, DT2 and TraiNet's developer indicate that it is being used solely to collect data required by USAID for reporting purposes and not as a tool to promote results￾oriented training. The Team observed that each TA Contractor visited had developed a separate data system to address its specific needs. For instance, two contractors visited needed to know the names, organizational affiliations and training program details (topic, date, duration, etc.) of in-country trainees, none of which could be captured by TraiNet. They designed flexible sys￾tems that could be adapted to the special needs of contractors embarking on significant in￾country training. The single largest user of DT2 services, IELP-II, developed its proprietary data management system from the inception of its project when it became evident that TraiNet was not appropriate. One contractor had heightened expectations when TraiNet was introduced that it could serve its internal needs, only to be disappointed to discover its limitations after it became operational. Even DT2, which manages TraiNet, designed a separate data management system that fit its needs in tracking myriad information about in-country, 3rd country and U.S. training. The abil￾ity of the DT2 Contractor to produce training data from its proprietary data management system in diverse formats and cross-tabulations greatly aided the evaluation team in analyzing critical information, very little of which was available through TraiNet. There has emerged two data collection systems in Egypt: one producing minimalist data on par￾ticipant training through TraiNet that produces no value added to the processor but fulfills USAID requirements; the other a series of customized data management systems that do, in fact, add value to the contractor. Of the many TA Contractors active in Egypt, only 13 are currently providing data via TraiNet on a regular basis to the DT2 Contractor. However, in view of the fact that the vast majority of par￾ticipant training is managed directly by DT2, most of the data is, in fact, being collected and re￾ported. This data is far more significant to the users (USAID/Washington and USAID/Egypt) than the in-country programs, which contain no information on the individuals trained or their institutions potentially affected. Moreover, the far greater per-participant cost for an overseas program would warrant closer monitoring in terms of impact. 5. Contractor management The table below illustrates the two distinct management styles that typified DT2 management before and after the evaluation conducted in January 1999. The first period included establishing the Cairo office with a traditional organizational structure and various procedures. There was a Page 24 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt heavy emphasis on designing, procuring and finalizing training needs assessments for each SO prior to undertaking participant training. The Early Period of DT2 Management (1997 – 1998) • SO/RP teams did not take the lead in assessing training needs sufficiently • DT2 took seriously its mandate to apply Best Practices • Organization into divisions each responsible for a different process and each in contact with all clients • Emphasis on "getting the process right" before embarking on training • Over-ambitious TNAs impeded implementation • TNA and other processes tarnished DT2-USAID-TA Contractor relationships and slowed disbursements • Confusion about whether DT2 or TA Contractors were to manage in-country training The second period followed a critical evaluation completed in January 1999, after which IIE in￾troduced some significant management changes. The evaluation stated that the rigorous applica￾tion of TNAs seriously interfered with the implementation of training. It was unacceptable to at￾tempt to institute Best Practices in this manner at the expense of moving the training forward quickly enough to facilitate the flow of USAID/Egypt development funds. After the evaluation, the emphasis quickly changed away from large Training Needs Assess￾ments towards facilitating training programs requested by customers (SO teams, TA Contractors and Egyptian partners). This approach whereby IIE sought to streamline training management in lieu of imposing a process thought to reflect Best Practices in training management was timely and well received by all DT2 users. The Current Period of DT2 Management (1999 – 2001) • DT2 Contractor responded to criticisms put forward in the evaluation • Training numbers increased dramatically • DT2 became more responsive to clients • The USAID/Egypt "pipeline" became unblocked • Management emphasized facilitating implementation over imposing Best Practices processes Evaluation and Options Development for DT2 Page 25 USAID/Egypt Aguirre International Findings • IIE has made significant progress in streamlining and rationalizing all of its implementing functions • There has been continuing confusion about the division of responsibilities between IIE and the TA Contractors regarding in-country training and some other areas • In order to be more customer oriented, DT2 has moved to a “reactive” stance in which it is not actively emphasizing implementation of Best Practices 6. Addressing USAID's Evolving Needs As USAID/Egypt responds to the dynamics of the country's political and economic situation, the Mission's programs evolve, affecting training interests and priorities in the process. Recently, USAID/Egypt has targeted geographic areas of the country for special attention, introduced three significantly large activities in the private business export sector, and introduced an SO that fo￾cuses on workforce development. The workforce development initiative alone entails five train￾ing programs, of which DT2 is procuring training for several. In addition, the Mission is now considering how to respond to overtures for USAID/Egypt to manage a large volume of overseas training at the post-secondary level. The Mission has needed DT2 to be sufficiently flexible to respond to these new initiatives. Both the private sector focus and the workforce initiative have benefited considerably from DT2's flexibility and expertise. Findings DT2 has enhanced the Mission's ability to respond to new initiatives, because it provides a cen￾tral fund which can be reallocated as needs change. At the same time, the funds that DT2 has available to manage training are fixed. The level of effort established for DT2's services was set at the beginning of its six-year term, based on the assumption that most training processes would follow a standard set of procedures. Yet IIE staff have responded fully to Mission requests to manage the new initiatives as well. The management intensity of some new initiatives has meant that the staff has been able to pay less attention to some of the regular tasks in its scope of work, such as follow-on and evaluation of training. The Mission will also need to think about building and sustaining the capacity of Egyptian or￾ganizations to provide training of the type DT2 has sponsored, once the Mission phases down. DT2 has begun to do this through its in-country procurements of training, but much more could be done. For example, TGP could include the provision of targeted technical assistance in or￾ganizational development delivered by performance consultants steeped in the methods used successfully in North America for over a decade and applied to both NGOs, government agen- Page 26 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt cies and the for-profit service industry with great success. Egyptian service providers (including training institutions) can develop the capacity to carry out many of the functions IIE currently manages for USAID/Egypt if a Sustainable Development Action Plan is developed and applied by the TGP contractor. a. Private Sector As discussed elsewhere in this report, training for the private sector requires management inno￾vation and attention to the needs of a special audience. In general, DT2 has been able to respond to many demands but is hampered by the labor intensity of some private-sector requests and the practices that prevail from a dominant focus on public-sector training. The most challenging types of private-sector training for DT2 to plan and manage are internships at U.S. firms, trade show attendance followed by commercially oriented visits in the United States and export-oriented 3rd country training. Last-minute requests and changes are typical with business participants and can cause havoc with DT2 resources. Careful monitoring of training results and on-site follow-on take on greater importance with some private-sector participants. The extent to which DT2, and its successor TGP, can develop business-friendly procedures and approaches will be instrumental in determining the effective￾ness of training investments made by USAID/Egypt in the private sector. b. Gender Under DT2 the Mission has improved dramatically its compliance with Agency guidelines re￾garding the inclusion of women in training programs. Data produced by IIE covering all training implemented to date indicate that roughly 30 percent of all participants in overseas programs – both short- and long-term – are women. This is probably a significant change from pre-DT2 years when TA Contractors were managing participant training directly, many of whom were not providing reliable data on gender. It is significant that the USAID/Egypt did not seek to artificially raise its level of participation by women in USAID-funded training by co-mingling in-country with overseas training, as do some Missions. Training data correctly disaggregates gender by training type in order to assess the degree of participation by women in USAID training according to investment rather than by training days. Despite its progress toward including women in training, the Mission should set a target of 50 percent for women overseas training participation, as do many other Missions. To more accu￾rately reflect progress, USAID/Egypt might also monitor gender in training investment by sector as well. To what extent, for example, has the contribution of IELP-II participants to the DT2 training pool increased the percentage of women trained, in view of the greater numbers of women in education? As the percentage grows, USAID/Egypt will need to track gender by sec￾tors to ensure equity across sectors, especially those in non-traditional women fields such as ag- Evaluation and Options Development for DT2 Page 27 USAID/Egypt Aguirre International riculture and business. As is often the case, reporting averages, rather than breaking down the statistics in more detail, tends to hide both the strong and weak elements being analyzed. Finally, if long-term academic programs take shape, DT2 and TGP will need to develop trans￾parent and well-publicized policies to recruit, and possibly even assist through remedial work and test preparation, women for university placement. Many solutions have been used in other countries to overcome cultural and educational constraints to reaching gender targets in long￾term programs. Both the DT2 and TGP Contractors will need to be abreast of these possibilities and ready to implement them in Egypt. c. Geographic Targets The Mission has established geographic target areas for emphasis in its development program but the Team was unable to determine the effect of these areas on the management of training by DT2. The DT2 Contractor is not generally involved in participant nomination and selection, the two aspects of training where geographic targets would pertain. The Mission has not sought to use DT2 to affect the overall emphasis of USAID/Egypt on its geographic target. To the extent that the Mission decides to begin tracking and measuring training investments in the future by geographic area, it will need to arrive at precise definitions which can be included in the TGP design. For example, what elements of a participant profile would determine the source geographic area – birthplace, current employer, residence? d. Future large long-term academic programs An increase in the use of DT2 funds for long-term academic training has occurred over the past year, in particular for MBA degrees targeting the private sector. In addition, a large-scale U.S. academic training has been discussed for several sectors, such as Information Technology. Sig￾nificant programs of 50 or 100 Egyptian participants must have proper recruiting, selecting, testing, placing, preparing, monitoring and re-integrating support to be successful and avoid high non-return or unemployment rates. They would require new pass-through funds coupled with additional contractor resources for management. B. Funding Mechanisms The six-year budget for DT2 totals approximately $120 million, including over $20 million for contractor administration of training activities and $100 million of “pass-through” participant costs, in support of the Mission’s various strategic objectives. This level of funding of some $15 million annually, while viewed as huge by many, in fact represents less than 3 percent of USAID/Egypt’s Operational Year Budget (OYB). In setting aside these resources for training, DT2’s planners did not intend to finance all Mission training. Rather, the $100 million was expected to pay for all overseas training and much of the Page 28 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt off-the-shelf in-country training. TA Contractors, it was assumed, would continue to provide di￾rectly or to procure with their own budget allocations, much of the tailored, in-country training. In this sense, the Mission simultaneously established a six-year $100 million ceiling for training that could be procured through the DT2 mechanism and a floor for the amount of training the various SO and RP teams would need to procure. Viewed from the perspective of TA Contractors, DT2 fragments the decision-making: con￾tractors receive one pot of resources that can be used for any type of input and a second pot that can be used for training only and must be administered through DT2. In this way DT2 limits the contractor's flexibility, since DT2 resources are not fungible with other non-training inputs. Because the implementation of TraiNet has not succeeded in collecting and collating comprehen￾sive data on TA Contractor-financed in-country training, it is not possible to determine the amount of training that is financed from TA Contractor budgets* . In the course of its interviews, the Team examined Mission programs that rely almost entirely on the DT2 budget for training and others that claim to finance as much as 90 percent of their training from the contractor’s budget. The allocation of DT2’s training funds. The Mission's CTO for DT2 has allocated the “pass￾through” training budget among the seven Strategic Objectives and approximately 40 Results Packages. He revises this budget periodically to reflect the changing needs of the program and the ability of SO/RP teams to implement planned training. At one point the consolidated plan￾ning budget grew to over $135 million after which that budget was reduced to $85 million as part of the Mission’s overall “scrubbing” exercise and to reserve resources for training needs ema￾nating from the Mission's new Country Development Strategy. With the subsequent addition of several private sector training activities and some upward adjustments in other activities, the cur￾rent budget climbed to $111.4 million. The fact that the budget is permitted to rise above the planned expenditure level reflects the judgement of the CTO and IIE that much of the training included in DT2’s approved Training Plans will not be completed before the 2003 completion date. Allowing this “over-budgeting” in effect sets up a desirable type of performance-based competition: those RP teams and TA Con￾tractors that can implement their Training Plans are rewarded, whereas those that fall behind may have to forgo funds for the others. Since total DT2 training expenditures by the completion date may be below the $100 million authorized, the clock rather than project managers may produce the cuts needed to bring the budget down to the authorized level. Thus, the competition seems not to be for resources but for using the allocated resources before time runs out. Again, since funding is not really rationed, the scrubbing exercise, which cut some $50 million from the projected DT2 training budget, essentially reduced the over-funding included in early * The term "contractors" used in this and other sections of the report includes "grantees." Although distinctions exist between these two terms regarding procurement and the relationship between USAID/Egypt and its "implementing partners," these are not germane to the training issues being treated in this report. Evaluation and Options Development for DT2 Page 29 USAID/Egypt Aguirre International budgets. It did not create a real funding shortfall that significantly limited the training available to the SO/RPs. The current allocations of DT2 funds to SO/RPs vary greatly in size (as do the size or RP budg￾ets): two DT2 allocations are under $50,000 and seven are over $5 million each. The Team un￾covered few complaints about the size of these allocations. The comments focussed on the per￾ceived difficulty TA Contractors have had obtaining approval for training activities outside the "approved" training plan. Unlike other Missions where a lack of budgetary resources would be the primary constraint, USAID/Egypt staff are more concerned about bureaucratic constraints to implementing training quickly in order to spend the available resources. While the level of funding is not an issue, the Team noted that TA Contractors view the “dual” or “split” budget arrangement, with most of their resources coming through SO and RP teams and a supplemental part from DT2, as somewhat problematic. Theoretically, a contractor could first use up all of its DT2 money and then, if training priorities were not met, supplement this with regular funding which is fungible across inputs. However, the situation is considerably more complicated with OYB and DT2 funding on different schedules and DT2 with its own set of procedures. These factors make it more difficult for contractors to plan an integrated budget based on both sources. Some contractors appear to utilize DT2 funding for less critical training and to implement key programs from budgets they control. Some regard DT2 as an extra source of resources to use as time and need arise, outside their regular activities. Several contractors learned about training allocations earmarked within DT2 precisely for their use only by accident. Some contractors indicated that data about their allocated portion of the DT2 budget and expen￾ditures are not readily available. Views about the process of changing budget allocations were not consistent. Two USAID officers viewed the DT2 budgeting process as quite flexible, capable of rewarding those who promptly used their resources and who sought supplemental funding. At least one contractor felt that the process to obtain additional resources is not transparent and not responsive. One USAID/Egypt RP manager feared that were training resources not protected in a central fund and were given to TA Contractors, far less training would be implemented. The Team’s interviews with TA Contractors suggest that the separation of DT2 funding – the fact that it is in another contractor’s budget and subject to additional rules and approvals – often leads them to feel little “ownership” in the DT2 training activities. This can result in their in￾vesting less effort in the planning, selection and follow-up processes and in less integration with the contractor’s technical assistance program. The lack of a sense of ownership is likely also to underlie what appears to be a lack of cost-consciousness about the use of DT2 resources among some TA Contractors and USAID technical officers. Such behavior with respect to training is likely to encourage lower priority training and to reduce the overall impact of training on the achievement of the Mission’s SOs. On the positive side, adept USAID project officers and TA Contractors who want to increase the amount of training in their activities have found that DT2’s central budget can be useful. Two Page 30 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt USAID officers noted that DT2 had been willing to reprogram funds from other RP teams that had left them idle. In sum, DT2’s centralized funding mechanism seems to have yielded the following advantages and disadvantages: • Advantages: Because SO/RP teams and TA Contractors did not feel much ownership in their DT2 allocations, it has been relatively easy to reprogram funds to satisfy new Mission initia￾tives and the needs of RP teams that use up their allocations and make the case for more. • Disadvantages: The DT2 funding mechanisms sets up a second resource allocation system that to some extent distorts resource allocation, favoring training over other inputs. Those re￾ceiving these allocations and responsible for seeing to their effective use have less flexibility in using them than they do with other funds. They often lack a sense of ownership over these resources and seem to put less care into their effective use. Evaluation and Options Development for DT2 Page 31 USAID/Egypt Aguirre International III. Key Recommendations and Options This Chapter consists of three sections: the first (A) summarizes the "pros and cons" of the DT2 experiment in centralizing training management and funding; the second (B) looks at the options available concerning the funding of training sponsored by USAID/Egypt; and the third (C) pro￾vides specific recommendations for each of the key management issues examined by the Team. A. Balance Sheet on DT2 USAID’s objectives in establishing DT2 were identified in Section I above. On the basis of its review, the Team concluded that DT2 a) has succeeded in bringing the management of training into compliance with USAID regulations; b) has made progress in streamlining training, al￾though further progress is desirable; and c) has not greatly influenced the degree to which USAID training is re-engineered or adheres to both the rule and the meaning of Best Practices. The Team’s conclusions in these areas are discussed in the following section and summarized in the Balance Sheet on the following page. Page 32 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Summary Balance Sheet Achievement of the Objectives of DT2 ASSETS / SUCCESSES Ensure Compliance with USAID Rules and Regulations The Mission’s training management system (of which DT2 is a principal part) ensures consistent observance of USAID rules and regulations related to participant training. Streamline Management of Training DT2 has reduced the management burden for training (especially participant training) on USAID/Egypt and on TA Contractors; Increase the Impact of Training on SO/RP Results By asking appropriate questions (e.g., about the relationship between proposed training and intended organizational changes), DT2 has helped TA Contractors and USAID/- Egypt staff to become familiar with Best Practices. LIABILITIES / FAILURES Ensure Compliance with USAID Rules and Regulations USAID/HDD/ET considers that high com￾pliance continues to warrant close Mission oversight. Streamline Management of Training Although DT2 has become more user￾friendly over the past two years and takes responsibility for a number of time￾consuming training management tasks, some TA Contractors continue to view the DT2 processes as difficult and time￾consuming and not infrequently prefer to do their own planning and implementation of training. Increase the Impact of Training on SO/RP Results Most TA Contractors seem to be identifying appropriate training that complements TA and contributes significantly to the achievement of USAID’s Results Packages. But it is difficult to find evidence that at￾tributes the reason for a better link between training and results to DT2. Moreover, training implemented through DT2's cen￾tralized mechanism is sometimes consid￾ered off-budget by decision-makers causing it to be less well integrated with TA and somewhat less cost-effective. 33 Ensuring Compliance with USAID Rules and Regulations With regard to compliance to USAID’s training rules and regulations (ADS 253), DT2 has reg￾istered some notable successes. Centralizing participant training with an outside DT2 Contractor has resolved most of the problems cited in audits and memos in the early 1990s. Although some inappropriate use of USAID's Invitational Travel mechanism to circumvent Mission training regulations and the Agency guidelines (ADS 253), the Team did not discover widespread abuse of this procedure as is the case in some Missions. This positive result is due largely to IIE's care￾ful attention to compliance issues and its close working relationship with USAID/Egypt staff di￾rectly responsible for training procedures. The CTO and the Mission training expert are to be credited for working hard to ensure that DT2 carries out its responsibilities effectively. They re￾view training events, back up contractor staff in its efforts to apply the rules, and process waivers painstakingly and often thanklessly. The CTO believes that this level of USAID oversight is necessary due to the past high turnover of the Contractor's staff and the lack of training and expe￾rience. Streamlining Management of Training As noted above, in its initial two years, DT2 procedures were widely perceived as obstacles, and the pace of Mission-supported training declined. The manner in which DT2 assessed training needs, a principal source of the delay and annoyance during that period, has been adjusted. The DT2 Contractor now relies largely on the TA Contractors, EPIs and USAID staff to identify training needs, and it offers several levels of assistance when professional help is sought. The contractor has also simplified its procedures and reorganized its staff in an effort to make using DT2 easier for collaborating organizations. These efforts have been quite successful, and DT2 is clearly much more “user-friendly.” IIE’s success in this regard is reflected in the greatly ex￾panded numbers of Egyptians trained through DT2 during the past year. At the same time, the assessment team believes that the DT2 Contractor could make further progress in creating a service orientation toward all of its collaborating partners. DT2’s efforts have also succeeded in lessening the management burden associated with training on USAID’s TA Contractors and on USAID/Egypt. Although TA Contractors continue to criti￾cize procedures and the time they require, many know that training, especially overseas training, is management intensive. Even those who complain often would not wish to take over the DT2 Contractor’s responsibility for competing and contracting with training providers or processing participants for overseas training. As is the case with many USAID Missions, a contractor (IIE) has taken over many responsibilities previously handled by USAID direct hire and foreign serv￾ice national staff. There is currently only one U.S. Direct Hire officer (the CTO), who will be replaced by a PSC employee, compared to over 8 full-time employees prior to DT2. Finally, DT2 has also served the Mission as a flexible mechanism to provide training in response to new initiatives. In this regard, both the available funding for training and the IIE staff have served as reserve resources upon which the Mission could call as needed. Because the $100 mil￾lion training budget was ample, addressing these new initiatives has not affected existing initia- Page 34 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt tives. However, the management burden on IIE in implementing these initiatives has stretched its manpower and management budget. Increasing the Impact of Training on SO/RP Results Measuring DT2’s contribution to promoting Best Practices is problematic. The initial contractor strategy, which was prescribed in detail in the DT2 contract, included rigorous training needs assessments and aggressive programs to promote other similar practices, most of which gener￾ated anger and resentment in the very audiences whose level of appreciation and commitment to Best Practices DT2 was supposed to advance. DT2 has been unable to persuade others of the need to link training to institutional needs, to execute meaningful stakeholder agreements, to do follow-up and evaluation, and so forth. Currently, needs assessments and development of train￾ing specifications are largely in the hands of TA Contractors, and to a lesser extent EPI personnel and USAID technical staff. The actual training is contracted by training providers. IIE does carry out evaluation and follow-up activities on a selective basis; these efforts to increase the impact of training already completed are independent of the activities of TA Contractors. Overall, the IIE's success in shifting its focus over the past two years to meet USAID’s demand that it facilitate and streamline training has limited the extent to which it provides professional advice to promote re-engineering and Best Practices in training. Because IIE has recently focussed on helping TA Contractors and SO/RP teams procure the training they request with fewer questions asked, it has paid less attention to Best Practices. This is not to say that DT2 and its partner institutions are carrying out inappropriate training and implementing training poorly. Most TA Contractors seem quite competent to identify training needs and to develop technical specifications, with the result that the bulk of training is appropri￾ate and would appear to contribute to the achievement of USAID’s goals. B. Funding The key funding issue for TGP is whether to continue with DT2’s system of centralized training resources, or, alternatively, to allocate those resources to SO/RP teams so that they might decide how much to spend on training. This section is intended to assist Mission management to weigh the advantages and disadvantages of these alternatives and to provide the assessment team’s rec￾ommendation. The Team recommends that USAID/Egypt decide to decentralize the funding of its centralized training management mechanism, TGP, taking into account the advantages and disadvantages of the DT2 experience with a central training fund. Details on how that funding process would function is found at the end of this section after the following discussion. A useful way of thinking about these alternatives is to consider what questions each requires be answered, by whom and at what time. The key questions posed by centralized and decentralized funding of training are summarized in the following table. 35 Key Budgetary Questions Posed by Alternative Mechanisms for Funding Training Centralized Funding of Training (training resources set aside as in DT2) Decentralized Funding of Training (all resources allocated to SO/RP teams) Mission Management How much of the Mission’s budget should be allocated for training? How much of the Mission’s remaining budget should be allocated to each SO? Mission Management How much of the Mission’s budget should be allocated to each SO? DT2 Project Officer How should training resources be allo￾cated among SO and RP teams? DT2 Project Officer SO/RP Teams and TA Contractors What training activities should be un￾dertaken with the TGP allocation? What share of regular budgetary re￾sources should be allocated to training and to other inputs to best achieve the desired results? SO/RP Teams and TA Contractors How much of our budgetary resources should be allocated to training and to other inputs to best achieve the results desired? The most basic budgetary decision that the top managers of each USAID Mission make is the allocation of program resources among SOs. Resources flow to SO teams then to RP teams and, in most cases, to TA Contractors and grantees.* A basic tenet of re-engineering is to empower those contractors and grantees with the authority to make budget allocation and staffing deci￾sions as needed, and then to hold them accountable for results.† * It is important to note here that the term “technical assistance contractors” in referring to USAID’s contractors is something of a misnomer. USAID’s contractor selection process normally begins with a Request for Proposals (RFP) that specifies that the winning institution is to provide a variety of inputs that include technical assistance and training. Competing firms often carefully build consortia of institutions to ensure that they can meet any potential training need as well as provide technical assistance. Consistent with these steps, USAID selection procedures assess competence in both areas. † Even before re-engineering, USAID resources flowed to contractors to implement most programs. However, pre￾vious systems were tightly controlled by USAID staff. In the words of a prominent management expert, USAID “over-planned and under-managed.” This criticism described a system in which USAID concluded detailed multi￾year plans, and then asked contractors to implement them as designed. In contrast, under re-engineering the principle Page 36 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt The centralized funding of training such as it exists in DT2 requires that Mission management answer a second question: how much of the Mission’s budget should be used for training? Al￾though this question may initially sound reasonable, there is no way that a manager can make that decision wisely before the activity period begins. This question should not be posed and cannot be answered appropriately a priori. Mission management will either set aside too much money, encouraging wasteful training expenditures, or too little money, encouraging the under￾use of training as a tool to achieve the Mission’s stated objectives. Rather, the determination of the share of total resources devoted to training (and to other inputs) should be the sum, ex post, of decisions taken by lower management levels over the course of some time period. In USAID/Egypt, training is the only input for which there are special set-aside funds. Mission management does not decide, for example, how much of the program budget to use for comput￾ers, roads, bridges, PVC pipe, technical assistance or travel. When decisions about training or about these other types of inputs are made for the whole program, they lessen the management authority of those lower down in the system who are responsible for producing specific results. They effectively limit the authority of those people by determining that, of the resources allo￾cated to them, at least a certain fixed amount must by use for training or for other inputs. The centralized funding of training is thus inconsistent with management principles that are central to re-engineering. As indicated in Section II above, the centralization of funding under DT2 did lead to a less effi￾cient use of resources than was intended. The Team therefore recommends that the Mission re￾vert to the more traditional decentralized approach when it designs TGP. Decentralization can be expected to yield benefits in the following area: • Administrative costs and efficiency • Avoiding duplicative and confusing allocation processes. Centralized funding of training removes those resources from the normal process of allocation of funds to SO/RP teams. It thus requires the establishment of a second process to allocate, administer and oversee training resources. This second allocation process is time-consuming and unnecessary, and it adds an additional burden to USAID officers at all levels. • Conflicting roles and responsibilities. Since the training funds are under the Training Of￾fice, it forces the training office staff to review the judgements of technical officers about the cost-effectiveness of individual training activities. The funding system thus dilutes re￾sponsibility and encourages controversy and delay. of "performance contracts” gives contractors the flexibility to move resources within an overall budget, to hold them accountable for results and to remunerate them according to performance against those results. 37 • Activities and focus of USAID training staff. The centralized funding of training diverts the time and attention of USAID training personnel from efforts to improve the quality and im￾pact of training to issues of allocation and funding oversight. • Efficiency. Decentralizing the funding of training enhances the ability of RP teams and TA Contractors and grantees – those responsible for results – to allocate resources to best achieve RPs. It thus improves budgeting and planning and makes better use of available resources. • Integration of TA and training. Decentralized budgeting of training encourages TA Contrac￾tors to take an active role in each of the substantive processes related to training, from deter￾mining training needs to following-up in the workplace. Thus, the decentralization creates in￾centives for contractors to integrate training and TA, i.e., to use those inputs to reinforce one another to maximize impact on results. Details on the Recommended Funding Option: The implementation of the decentralized funding mechanism will be similar to what is done in many other USAID Missions. A Mission selects a central contractor from the pre-competed central IQC (previously Global Training for Development, which will be replaced in June 2001 by START) through a mini-competition based on the TGP Scope of Work. That contractor opens a Cairo office to manage training as described in the SOW. That operation is entirely un￾der the supervision and contracting authority of USAID/Egypt. The contract is funded by each SO through SO-based Task Orders. Each Task Order can be negotiated with the contractor and includes the contractor support costs needed to implement the training described in the Task Or￾der. An SO might have one or two Task Orders per year. Task Orders can be general enough to allow for modifications during the year in terms of the specific training to be implemented. If the Mission wanted to fund some of the TGP core costs for certain cross-cutting activities or targets of opportunity training not directly related to an individual SO, it could issue a Task Or￾der directly. Or, were the Mission to fund a cross-cutting long-term academic training program, it could issue a specific Task Order for that program without going through an SO. Alterna￾tively, it could issue a single Task Order with multiple SO funding. All of the above methods have been used by other Missions with the GTD contract and would be possible with the new START mechanism. C. Recommendations Concerning the Principal Management Issues All of the principal issues addressed in this section concern the management of future USAID/Egypt training rather than the funding. This section treats the specific management is￾sues below each with short-term and long-term recommendations. Background on the rationale for each recommendation is found in the discussion on each issue in Chapter II. The management Options available to the Mission in designing TGP are similar to the options available to the Mission in 1993, with some changes: Page 38 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Options (1) Centralized management of most training funded by USAID/Egypt implemented by an outside contractor, as has been done under DT2 (2) Decentralized management of all training to TA Contractors, reverting back to the previ￾ous system but without heavy involvement of a USAID Training Office (3) A mixture of centralized management for certain types of training and decentralized for other types. This report has addressed many management and funding issues that affect the choice of the above management option. In consideration of all of these issues, the Team strongly recom￾mends Option (1) where USAID/Egypt continues with the centralized management by a U.S.- based contractor of most of its training programs, whether in-country or overseas. The DT2 ex￾periment has been largely successful which would not argue for reverting to a system confiding important administrative decisions either to sector-specific TA Contractors, some without train￾ing management capacity, or to the USAID Mission itself, which cannot increase its staff or management burdens realistically in the current reengineered environment. Option (2) cannot be considered due to the inability to ensure standardized approaches to training processes and minimal requirements to ensure Best Practices with so many independent con￾tractors each handling training differently. Even were several sector-oriented TA Contractors to be considered as training management specialists for their sector, USAID/Egypt could not ensure that Best Practices or the performance improvement aspects of the management of training would be properly addressed. The management of training of the magnitude of that found in the diverse portfolio of USAID/Egypt is best left to organizations with expertise in handling large numbers of individuals, partner institutions and training providers. Option (3), if clarified carefully, could be considered. However, in selecting Option (1), the Team would include in its design prescriptions for TGP, as stated in this report, the provision that TA Contractors have flexibility in determining which organization (the TA Contractor or the TGP Contractor) would handle designated management tasks and responsibilities. Option (3) would imply a pre-emptive assignment of management tasks to either a central contractor or a sector contractor. Another dissuasive argument is that increasing amounts of training are cross￾sectoral, such as "NGO management strengthening," "financial management," or "computer pro￾gramming." The following section supports the choice of Option (1) while addressing the findings in Chapter II concerning specific aspects of the management of in-country and participant training. Each section below contains both Short-Term and Long-Term recommendations. The former are changes that USAID/Egypt and IIE can consider implementing prior to the end of DT2 funding; the latter are aspects to consider including in the design of TGP, the future "post-DT2" activity. 39 Assessing Training Needs Short-term: • In light of the shift over the past two years from promulgating Best Practices by rigorously applying administrative procedures (e.g., TNAs) to managing training outputs, IIE and the CTO should review the human and financial resources available in DT2 to consider a more aggressive role as a catalyst for improving the way organizational needs are diagnosed. • Within its present budget, however, IIE should begin familiarizing TA Contractors and other Egyptian clients (e.g., training providers) to the existence and applicability of modern tech￾niques for determining organizational training needs, such as short workshops on rapid methods for work unit performance appraisals. • DT2 should review its responsibilities in ensuring that training needs are appropriately de￾termined and settle on a workable approach that ensures a minimum level of needs assess￾ment for all training that also takes into account the needs of the users of DT2 to have their programs implemented. It may be helpful to distribute concise written guidelines that sum￾marize this approach and illustrate the information and survey techniques a TNA should in￾corporate. • Common sense should continue to be used when deciding whether to update an existing TNA; the users of DT2 should not be obliged to fulfill a bureaucratic requirement (to update a TNA) but should be engaged in a process of assessing whether and how training can lead to organizational change. A single meeting may suffice to "update" the TNA sufficiently. Long Term: • TGP should have the mandate and resources to provide high-quality technical and practical advice to SO/RP teams, TA Contractors, Egyptian consultant/trainers and partner institutions to promote the widespread use of contemporary organizational performance analyses by Egyptian firms, NGOs and government entities. Over the last decade virtually all successful organizations have adopted principles espoused by the American Society for Training and Development, the International Society for Performance Improvement and others to link training to work unit performance improvement. Future training sponsored by USAID/Egypt must move from implementation to performance improvement with increasing participation and support from leading Egyptian training providers. The TGP mechanism offers an ideal venue for this level of targeted assistance. • Since the initial demand for such assistance may not be high, as is often the case in develop￾ing country work environments particularly fearful of change, the "Performance Improve￾ment Advisor" working from the TGP Contractor would be responsible for creating demand Page 40 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt for human performance technology that is adaptable, cost-effective and appropriate to the Egyptian context. Planning and Processing Training Short Term: • USAID/Egypt should allow the TA Contractor and IIE to amend the Training Plan as needed without specific Mission approval, as long as the new budget does not exceed what was pre￾viously approved. This change would help speed up training starts, especially for ad hoc programs, and put the responsibility squarely on the DT2 and TA Contractors to justify the changes made. In this regard, USAID's relationship should evolve from "gatekeeper" and "watchdog" over training funds to "results-monitor" of training implemented under DT2. • To the extent permitted by Agency-wide guidelines (ADS-253), the USAID/Egypt Office of Human Development and Democracy should continue its efforts to streamline travel regula￾tions and to revise waivers to be in line with Mission programmatic objectives, in collabora￾tion with the DT2 Contractor that is charged with complying with, and implementing, proce￾dures mandated in the Mission Training Order. • Since the private sector has particular processing and logistic requirements that call for solu￾tions distinct from those designed for public-sector clients, DT2 should consider establishing a modified set of guidelines specifically for business participants on overseas programs. The working relationships, and perhaps even specific travel regulations, should be fine-tuned to enhance a strong customer orientation within DT2 that addresses private-sector needs. For example, DT2 might consider providing limited travel or logistic assistance to business peo￾ple recommended by TA Contractors who need to attend trade shows or conferences at their own expense. In such cases, the TA Contractor might pay a "processing fee" to the DT2 Contractor for stakeholders in the business community with sufficient financial resources available. • While the DT2 Users' Guide presents current regulations in a clear manner, it appears that DT2 could circulate the Guide more widely to reach everyone who needs the information. Long Term: • Give TA Contractors the option of procuring TGP services directly, such as for in-country training or customized U.S. short-term training, via a "fee for service" arrangement. In some cases the TA Contractors are better equipped and positioned to manage training than the centralized contractor. To avoid pre-empting these decisions, include in the TGP design pro￾visions for flexibility. For overseas training, a TA Contractor might procure TGP's visa, ori￾entation, medical clearance and travel services only (for a set fee) and handle the U.S. ar￾rangements directly. Another TA Contractor might opt for the full range of TGP services, 41 for example for a U.S. OST. However, all long-term U.S. academic participants would be handled by a central U.S. placement contractor. • In the future, the TGP Contractor can assist TA Contractors acquire the training expertise needed to provide sufficient information about their specific training needs to the TGP Con￾tractor. TA Contractors with significant training in their portfolio should employ a training specialist to work directly with a counterpart at TGP. For TA Contractors with less training, TGP could offer more services in planning, training design and follow-on. In cases where the TA Contractor or Egyptian partner institution does not see the need for these training services, the TGP Performance Improvement Advisor would play a role in promoting the ad￾vantages – both in terms of return on investment and organizational change – from applying Best Practices. Monitoring, Follow-on and Evaluation Short Term: • Follow-on in the workplace should be provided, at a minimum, for all long-term training, both in-country and overseas, and selected short-term U.S. programs. To accomplish this, IIE should consider reallocating staff resources and considering new, less labor-intensive techniques. Criteria should be established, according to the length, cost and intensity of overseas training, to select short-term participant programs for follow-on support at the work place. • IIE should review the respective responsibilities and tasks of its evaluation and follow-on staff to find ways to integrate the two functions more closely. • IIE should undertake an appraisal of the use of information gleaned from Monitoring and Evaluation reports, in collaboration with TA Contractors and other users, to determine ways to increase their value, impact and utility. • In order to further strengthen local capacity and provide for greater long-term sustainability, consideration should be given to provision of at least parts of DT2’s special Follow-On pack￾age to long-term in-country trainees, as resources permit. Long Term: • Based on the experience of DT2 over the next two years, the TGP design should clarify the type of training evaluation needed in order to support organizational performance improve￾ment rather than be limited to fulfilling an administrative requirement. • Under TGP some sort of follow-on support should be offered to selected in-country trainees, as a way to develop local capacity and enhance sustainability. Page 42 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Training Data Collection and Management . Given the volume of in-country training implemented by TA Contractors in Egypt in subjects and fields that run the gamut, is it useful to collect all the data with no order of magnitude of pri￾ority? Short Term: • Continue with the same arrangement by which DT2 manages TraiNet • The precise need for data collection of in-country training should be clearly established prior to launching an effort to improve its collection and management by TA Contractors • IIE and USAID/Egypt should make another attempt to communicate with the TA Contractors to obtain better compliance regarding in-country training data • Given the volume of in-country training implemented by TA Contractors in Egypt in subjects and fields that run the gamut, USAID/Egypt and IIE should establish criteria in addition to the length of training (currently set at 3 days) for which training data will be collected. Long Term: • Review the use and need for data carefully when designing TGP so that information is not collected for which there is no local interest • Consider new possibilities of system compatibility between TraiNet and the others in use in Egypt Contractor Management The overall recommendations regarding contractor management of training are: Short Term: • Clarify some of the management functions with customers so that everyone knows exactly the organization with primary responsibility over specified tasks • Begin the shift from a strategy of quality assurance to an advisory strategy, within the limits of the DT2 contract and resources, by promoting better understanding of Best Practices among customers (periodic workshops, information dissemination, etc.) 43 • Give the DT2 Contractor more flexibility to adjust its staff levels and capacities according to the diverse types of training requests from TA Contractors and SO/RP teams. • Although IIE staff have established much improved working relationships with most SO/RP team members and TA Contractors, the professional dialog should be further strengthened to address lingering suspicions about DT2 that date from the period prior to the first evaluation. Informal get-togethers or structured meetings with selected SO teams, RP managers and TA Contractors should be scheduled to promote good relations. • In view of the upcoming departure of both the CTO and the Director of Education and Training, consider transferring the oversight responsibility for DT2 to the Program Office which is a neutral location better suited to oversee multiple funding to a central contract by different SOs (a solution used by other Missions). The new CTO could work out of the Pro￾gram Office in managing DT2 for all SOs. Long Term: • Delegate clearly the responsibilities for training management in the TGP design drawing on the experience of DT2 to date. • Base the assignment of training management responsibilities among the various organiza￾tions (USAID/Egypt, TGP Contractor, TA Contractors, Egyptian training providers, etc.) on the premise that users of TGP services should have choices • Design TGP to provide a mix of services that includes processing participants for training as well as enhancing the context for institutional performance improvement. • Include as one of TGP's objectives the strengthening of organizational capacity of Egyptian training providers • When TA Contractors or SO/RP teams choose to use TGP's services to procure tailored training programs, USAID/Egypt should have the flexibility to adjust TGP’s level of effort as needed to accommodate requests that take more than the normal effort. Supplemental Management Recommendations There are three additional management recommendations that were considered of such signifi￾cance that they deserved supplementary discussion in the body of the report: (1) Reassign management functions (2) Develop advisory services (3) Build the organizational capacity of local training organizations These are addressed below in greater detail. Page 44 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt (1) Reassign management functions The training management functions for most SO/RP teams can be divided into six categories: • Needs assessment and development of a medium-term Training Plan • Annual plans and budgets • Training program design (course specifications) • Procurement • Processing of overseas training • Monitoring, follow-on and evaluation In some cases, TA Contractors have training funds and manage many of these functions for the SO/RP team. In other cases, TA Contractors have no training funds and, in collaboration with DT2, agree to take primary responsibility for some of these responsibilities. For the TGP design, the Team recommends that the each of the management functions identified above be clearly put forward as an option for one of the following entities to manage: the SO/RP team itself, its TA Contractor, the TGP Contractor or the Egyptian Partner Institution. With the funds being managed at the SO/RP level, as the Team recommends, the training management functions can flow to the most competent organization to manage. The premise to be guarded in the future is the flexibility for the SO/RP team to decide the best responsible authority for vari￾ous training management tasks, within certain parameters (for example, that long-term partici￾pants be handled by the TGP Contractor). The Table below indicates the allocation of these functions to either the SO/RP team with its TA Contractor ("RP/TAC") or to the TGP Contractor ("TGP"). 45 Allocation of Future Management Functions for TGP Functions Management Responsibilities RP/TAC TGP Needs Assessments X X Annual Plans and Budgets X Course Specifications X X Procurement Tailored X X Off-the-shelf Overseas X X Off-the-shelf In-country Training X Processing of Overseas Participants X Monitoring, Follow-on and Evaluation X X The Table shows the SO/RP team and its TA Contractor would have the choice of using services provided by the TGP' Contractor for all but three management functions. The exceptions are a) annual plans and budgets, in which the team and contractor must take the lead, although they could request TGP's assistance; b) off-the-shelf ICT, for which TGP's procurement would con￾tinue to build the organizational capacity of local training providers by offering opportunities for high-volume business (see (3) below); and c) processing of overseas training, in which the TGP Contractor would have special expertise and Mission reporting and compliance responsibilities. The reallocation of management responsibilities above would effectively challenge TGP to be￾come customer-focused, as many of its services would be optional and requested by the TA Contractor only when needed. The TGP Contractor staff would have the flexibility to work more intensively with those SO/RP teams and TA Contractors that desired its services, thus creating opportunities for more customized services and greater satisfaction among its customers. This change would also reduce friction between TGP and TA Contractors that resent being forced into a centrally managed system. TGP could have two branches serving USAID/Egypt training needs: A processing service to continue management of overseas participants to maintain USAID/Egypt compliance and qual￾ity, and an in-house administrative and consulting services to respond to USAID/Egypt, TA Contractors and Egyptian partners with a range of optional planning and procurement services, many of which could be procured on a fee basis. The Team recommends that the contract for TGP implementation provide core funds to continue the processing service and to initiate and build up the in-house consulting services. Were the services offered valued locally, the TGP contract would eventually attract additional funding from SO/RP teams, TA Contractors, Egyptian organizations with training funds and international donors seeking training management services. (2) Develop Advisory Services Page 46 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt The Team recommends that IIE begin now to shift from a directive strategy to an advisory strat￾egy. The current strategy intended to ensure quality training based on Best Practices rests on the premise that if each SO/RP team submits to a set of standardized forms and procedures (needs assessment, plan, specifications, evaluation, etc.), then training programs will be of uniformly high quality and lead to Level 3 or 4 impact in their institutions. Yet the DT2 implementation experience has shown that imposing administrative procedures, albeit well intentioned, did not result in better training. Worse, the burdensome processes attempted by IIE in the early years resulted in clogging up the pipeline and irritating DT2 customers. Later modifications resolved that problem by loosening procedural requirements, but IIE now has less input into decisions about training that affect its quality and usefulness. It should start now to move toward the full￾fledged advisory service capability that should be a part of the TGP design. The TGP design should build on IIE's experience in promoting selected advisory services that further the development of quality training needs assessment, plans, specifications, follow-on, and evaluation, over the next few years. TGP staff could modify how it interacts with SO/RP teams and contractors. • First, it could develop brief written forms that help teams and contractors think through their management of each function: assessing training needs, developing training plans, writing program specifications, following up with participants back on the job, and evaluating train￾ing programs. For example, the Training Needs Assessment has never been formally modi￾fied, so TA Contractors often opt to minimize this step. In contrast, IIE has helped some teams and contractors assess needs and use these to write training plans. It could document this process briefly so that other teams and contractors can also benefit from it. • Second, IIE staff could have more frequent conversations with teams and contractors about these functions and what improves the quality and usefulness of training and how they might change their practices to get better training. Conversations could take the form of one-on-one dialogue, meetings, workshops, seminars, and other such events. The purpose of each one would be to expose more teams and contractors to good practices in training and to help them adopt these practices in their own activities. • The development of written forms and the conversations with teams and contractors should go hand in hand. Teams and contractors should be asked to use forms, and IIE should help them customize forms to their own activities. The purposes of forms should be to help train￾ing managers think about what will improve training, not to add documents to the files. Forms that are not useful should be dropped. This shift to an advisory strategy will have implications for TGP staffing. It will require TGP to employ more training specialists who can offer professional advice. The shift might also begin now, within IIE, if more training expertise can be ensured, through either short-term consultants and/or other mechanisms. Current IIE staff who are training specialists could then be relieved of 47 some of their present tasks to take on an advisory roles, and other staff members could learn to perform some advisory functions through workshops and/or on-the-job counseling. (3) Build the organizational capacity of local training organizations The TGP Contractor should provide services to local training providers that help them develop their organizational capacity and readiness to address the training needs of USAID and other lo￾cal organizations. Sustainability should be built into the TGP design so that local organizations increasingly assume the training management responsibilities currently handled solely by U.S. contractors. Core funds should be provided in TGP to promote local institution strengthening. As a result, USAID training done in Egypt should improve in quality, as local training organiza￾tions improve their practices, and it should become sustainable, once USAID reduces its program in Egypt. As a result of these changes, training should become more integrated with technical assistance, as TA Contractors take more responsibility for planning these two kinds of support in tandem. USAID's training system should become more informed by Best Practices, as advisory services replace mandated, intricate administrative procedures. And local capacity to manage training should grow, as TGP turns some of its attention to assisting training organizations. Addressing USAID's Evolving Needs TGP requires a contracting mechanism that allows it to adjust its level of management effort on a regular basis to adapt to USAID's evolving needs and priorities. This mechanism would, ideally, be put in place immediately for DT2, so that its staff can better respond to changes that are al￾ready evident. In any case, it should be built into TGP in anticipation of demand for training be￾yond the current horizon. Page 48 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Annex A Case Studies and Criteria for Selection 49 Annex A Case Studies and Criteria for Selection The group of Case Studies selected were to include … • One from SO-16 (the largest and most diverse SO) • One where virtually all of the training had been completed • One that had training of a significant size • One that had a large TNA conducted by IIE, in terms of cost • One that had a small TNA conducted by IIE, in terms of cost • One where a recently modified TNA had been used • One from the private sector • One using tailored or off-the-shelf in-country training • All training types (ICT, participant, long-term/short-term, third country) were covered • High and low Level of SO/RP reengineering covered • High and low perceived impact covered • One that involved an Egyptian institution or work unit • One that had a complex training program (several types of training used) The Case Studies were divided into two primary and secondary. Team members were to devote sufficient time with the primary Case Study to gain an in-depth understanding of the dynamics of training management and the relationship between IIE and the TA Contractor as possible through interviews and site-visits. As time permitted, Team members were to become familiar with the secondary cases. Primary Category SO-16 ALEB – Agriculture-Led Export Business SO-18 EUM - Egypt Utilities Management SO-19 CAIP - Cairo Air Improvement Project SO-20 HMHC - Healthy Mother Healthy Child Secondary Category SO-16 APRP (water) SO-17 MDI - Management Development Initiative SO-21 NGO Services Center SO-22 IELP-II – Integrated English Language Project II Page 50 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Annex B Training Needs Assessments – Supplementary Information 51 Annex B Training Needs Assessments – Supplementary Information DT2 personnel have done an informal study relating their evaluation results ratings of training activities with the type of TNA done. The lowest overall average result ratings were found for the 33 activities in which the early, lengthy TNA’s were done. Next lowest were 12 activities in which there was no TNA. The next highest were 2 using the “key informant” system (i.e., inter￾views or focus groups), and 73 activities based solely on training requests from the TA and/or EPI. Several hypotheses may be made regarding these highly informal comparisons. One is that the reason for the poor results achieved with the lengthy initial TNA’s could be not only their uneven quality, but the fact that the process was so lengthy that very little training was actually done before they became outdated. Another probable reason is that in some cases TA’s working closely with EPI’s may have assessed training needs quite well without going through any formal TNA process, or at least none for which a written record is available. It should be made clear that neither DT2, the USAID Training Office nor this team know with any certainty how training requests under this “Low Option” model are determined. Interviews with TA contractors suggest that in most cases these requests are based on a serious commitment to USAID-mandated per￾formance objectives and on an intimate familiarity with the technical and managerial constraints that both TA and training are to address. • Effective needs assessments can be done quickly and well, particularly if those doing the as￾sessment know the institution involved and its problems. However, few cases were observed in which modern rapid appraisal methods were used to assess needs. • Omission of key characteristics of a TNA may result in the following problems: o When training is based almost entirely on a list of requests from the EPI, there is a danger that training may benefit the participants individually, yet not yield the results sought for organizational change o When training is based on general assessments of the type of training needed in cer￾tain sectors worldwide, there is a risk that the training provided might not be suffi￾ciently focused on the needs of actual trainees in EPI's o When no TNA is done, the evidence suggests that unnecessary training may be done; that is, training may be done when training is not the solution to the problem. It is clear that TNA’s must be updated or redone periodically. Somewhat paradoxically, the need for updating arises most quickly when training is ongoing and effective, since organizational change is taking place more rapidly in such cases. Three types of TNA, referred to, respectively as “Low, Medium or High Option” in the IIE DT2 User’s Guide: 1) The EPI or the TA presents a list of training needs to DT2 and these are devel￾oped into a training plan; 2) If the EPI and/or TA Contractor are not sure of what is needed, DT2 Page 52 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt staff or consultants may go in to interview key informants and run focus groups, and 3) If there is real uncertainty about training needs, a consultant may be brought in specifically to engage in a more thorough TNA. At this time, most training is based on the first model, with the TA Con￾tractors usually taking the lead in assessing training needs. Other observations concerning TNAs: ! IELP-II completed an assessment of training needs for Ministry of Education (MOE) super￾visors in February 1999 that accomplished the key objectives of a "full-blown" TNA in only 10 pages, providing the information needed to plan and design training sufficiently. ! For Egyptian Utilities Management, three TNAs have been done at one or two year intervals since the initiation of the project. After the first TNA that indicated that it was premature to conduct an in-depth organizational assessment, targeted focus groups and interviews were organized the second time around inside the partner organizations. This led to a Training Plan that was useful. A third updated TNA was then conducted for which key interviews were held with staff at USAID/Egypt, TA Contractor and Egyptian Partner Institutions. A Training Plan incorporating the information collected was then developed during a meeting with all the stakeholders who had been individually interviewed. This entire process, which spread over 12 drafts of the proposed Training Plan, required 3 months but yielded a realistic Training Plan that had the support of all stakeholders. It should be noted that neither of last two "TNA’s" became a written document. Indications are, however, that the training eventu￾ally organized was closely linked to specific results. As the process unfolded, each TNA ex￾ercise uncovered new priorities and needs. ! In another case, no specific TNA was done, although interviewees mentioned that they used existing assessments done by large organizations such as the World Bank or UNICEF. It is possible that some pipeline problems noted in this activity might be attributed to the lack of assessment of KSA actually needed in key EPI. ! In some cases no TNA was done. Observations indicate that in at least some of these cases, the target groups of trainees were so broad that is was really impossible to link training to on the job performance. It is possible that in some cases the lack of a TNA led to unnecessary training; i.e., measures other than training may have been needed to improve performance and achieve the results sought. 53 Annex C Quick Reference Guide to Results-Oriented Training QUICK REFERENCE TO RESULTS-ORIENTED TRAINING Best Practices Abridged for USAID Strategic Objective Teams Prepared By: Andrew C. Gilboy Consultant Global Bureau Human Capacity Development Center United States Agency for International Development April 1998 USAID/G/HCD: Quick Reference to Results-Oriented Training Page 1 How SO Teams Can Use Best Practices to Get Results Frequently Asked Questions and Answers Š Š Š Š Š Evaluations of training impact have shown the correlation between getting results and planning ahead. As teams work on their results framework (RF), strategic objectives (SOs), and results packages (RPs), they should be thinking HRD at each step along the way. Human resources development, and the vehicle we call training, remain by far USAID’s most widely-used development intervention. Every team will use training. Whether our activity is “technical assistance” or “training,” the major output is the transfer of knowledge, skills and attitudes (“KSA”). SO teams must therefore plan how they will reach their IRs – and how they can find out whether impact is occurring – before beginning a training activity. If the SO team and its partners clearly identify the institutional performance gaps to address through training and understand from the outset the results aimed for, the indicators will already be established. That means it will be possible to monitor results along the way – something every team (and partner) needs to do. In short, planning training strategically ensures that each HRD activity will be linked to the team’s IRs and limited training dollars will get the best return on investment. Q: I know that planning is needed, but our team members simply do not have time. How can we plan ahead when we are constantly behind?! A: It’s a matter of time management and risk assessment. It is proven that if you invest time up-front to plan (and design) training correctly, you will reduce time spent later on trying to fix things or, worse, trying to discover if the training helped the team reach an IR. Why not plan in advance and reap the benefits later? Plan Training Strategically Q: How can I collaborate with partners when they do not know what they need? Isn’t this a nice concept but in reality a waste of time? A: The better question to ask is what is likely to happen if you don’t work closely with partners in planning, implementing and monitoring training. Survey after survey of training impact has shown that USAID increases the chances of obtaining organizational performance change when the targeted institution works side-by-side throughout all aspects of the program. If you want results, work with your partners, even if it takes more time, and especially if you think they don’t know what they need! Collaborate With Partners USAID/G/HCD: Quick Reference to Results-Oriented Training Page 2 By including partners from the onset, everyone will agree to the expected outcomes from the training program, understand the indicators to measure and decide who will monitor progress. There are several “types” of partners that teams will encounter, among them: a) local institutions the training is expected to benefit and where performance change will hopefully affect the IRs; b) local training providers selected to plan and manage training and produce results; and c) USAID’s U.S. institutional contractors (either local or U.S.-based) that may be closely involved in implementing RP activities. Because SO teams do not typically implement and monitor training directly, they rely on the expertise of their partners to produce results that can be attributed to the SO team’s investment. Moreover, to achieve a degree of sustainability, the team must share ownership for the training activity with its principal partners. As part of the development of the RF, teams get to know the organizations active in the mission’s priority sectors. They often help formulate USAID’s SOs and IRs. Once the CSP is approved, teams identify activities to achieve the IRs. Teams analyze key sectors and describe the roles of partner organizations in delivering the needed “services” to the people. This leads into RPs, through which most SO teams have passed, and equips them well to discern the precise institutional performance improvements needed to support changes in the sector. The team must conduct a performance analysis of every institution it wishes to assist. This investigation identifies the difference between the what the organization is producing now and what the team, and the institution’s leadership, want it to produce in the future. The “performance gap analysis” can then be used to determine what activities will lead to the performance changes needed to move, however slowly, toward achieving an IR. The team will need to determine whether training, technical assistance or commodity procurement, or a combination of these, will address the performance gaps identified. For example, if an SO’s partner institution could produce and disseminate price information on two food crops on regional radio weekly Q: Why shouldn’t our team simply offer training grants to our partner institutions and get on with it? We know the organizations need strengthening, and we do that by offering training to qualified employees. So where’s the big change from the old days? A: When you focus training on individuals, you reduce the likelihood of getting results that will lead to progress with your IRs. All long-term sustainable development has occurred through institutional change, not through haphazard training of people who “deserve” help, however well-intentioned. Of course, we train individuals, but only when their acquisition of knowledge, skills or changed attitudes will lead to performance improvements at their institutions. Otherwise, the SO team members will be Target Organizational Performance Improvement USAID/G/HCD: Quick Reference to Results-Oriented Training Page 3 Q: What’s the difference between the “performance gap analysis” [described above] and a training needs assessment? A: The SO team first needs to know what it wants a key partner organization to do in the future that it is not doing today. This concerns the organization’s output or productivity – what it does to help a team achieve an IR. The training needs assessment (“TNA”) hones in on the organizational unit where you have decided training is needed. Its purpose is to identify gaps or constraints in KSA (knowledge, skills and attitudes) that training can resolve. The TNA will lead directly to the design of a program, selection of trainees and Training Needs Assessments rather than quarterly, the SO team estimates that both supply and demand for the commodity will increase (thereby addressing one of the IRs). An institutional assessment will reveal why the institution is not performing at the desired level. The team (either on its own or drawing on training technical assistance from GTD or a local contractor) will then decide what change will resolve the problem. Does the institution need new computers, training for the staff to use existing computers, or TA. (Be aware, however, that the delivery of TA is often “training” in that the technical advisor transfers skills and knowledge to counterparts.) Before any training is agreed to, the SO team and its partners should know exactly the organizational performance constraints that the training will alleviate. If training cannot resolve the performance problem the team has identified, don’t train! If training can provide solutions, consider training but ask the critical question: what performance improvements are we seeking by offering training to this institution? This is indeed the “big change” (as asked in the box) from earlier days when USAID “allotted” graduate degrees to institutions that proposed candidates in their annual “training plans” with no indication how those coveted degrees would change the institution’s performance. The training needs assessment parallels or follows the performance gap analysis. Neither needs to be complicated or lofty, and both can often be performed with little or no outside assistance. The SO team may decide that it cannot effect change through training for the entire institution (e.g., a ministry) but that an improvement in the output of several key divisions (for instance, research or data collection) may yield the desired change. In this case, the SO team and its partners target an organizational unit in the institution for training assistance. A TNA is then conducted for that unit. Detailed information on how to conduct a TNA is found in the Best Practices guide. Remember that the purpose of the TNA is to identify and analyze the KSA deficiencies in the targeted organizational unit and to suggest how these deficiencies might be addressed through training. The methodology is standard and used by most corporations and organizations to discover what training is needed. USAID/G/HCD: Quick Reference to Results-Oriented Training Page 4 Q: Can we leave the selection of trainees up to the partner institution? If we believe in empowerment, why does the SO team have to get involved with selection? A: Yes, you can if Best Practices have been followed. If your team has already worked closely with the partner in identifying the training needed, your role in selection can be minimal (such as helping decide on criteria for selection or reviewing candidate profiles). Selection becomes almost routine if the proper background analysis and planning have been done. However, since the SO team is accountable for results, it will want assurances that the partner institution conducts itself appropriately . Are your partners selecting the appropriate trainees with the ability to learn and replicate training to others? Are the employees in the targeted work unit being selected? Empowerment? Yes, but with some preconditions. Select Trainees with Potential to Initiate Change If the SO team members were clear as to the causes for performance problems in the institution (as a whole) or at the level of the organizational unit, the TNA will flow easily. However, teams should remember that some causes for performance failures, either internal or external, cannot be resolved by training (such as non-payment of salaries, lack of a management environment conducive to improvement, civil unrest, etc.). Training can only transfer knowledge, skills and introduce attitude changes. A TNA begins its analysis at the institutional level (inspired by the performance gap analysis already completed) and works toward determining skill deficiencies at the employee-level in relation to job descriptions. It does not have to be elaborate. Rather, a solid TNA should clearly identify the skills or attitude changes that specific employees in the targeted organizational unit need to improve their productivity. Again, ask common-sense questions to help focus the TNA: “what should this unit be doing that it is not currently doing (the “performance gap”)?” and “what do the employees in the unit need to know to close that gap to increase the unit’s output, productivity or service quality?” The training needs assessment will answer these questions and identify the specific KSAs to be transferred to the trainees. Armed with this information, the SO team, in collaboration with partners in the institution, can plan training strategically. They can now easily write a Scope of Work that includes, a) a description of the performance problem to be resolved through training, b) an objective for the training showing the link to an IR and c) specific skills or attitude changes that need changing. This SOW becomes, in fact, the RFP and contract for a training provider to design and implement the training. It is self-evident that even if all the other Best Practices have been faithfully applied, and the selection of the people to be trained is flawed, the results anticipated from the training may not occur and performance changes targeted not achieved. USAID/G/HCD: Quick Reference to Results-Oriented Training Page 5 Q: In the past, USAID funded so much training and we wonder if it all brought about any change. How can we be sure to get results from training even if we follow all the Best Practices? A: Many USAID employees ask the same question. Much of the failure of training in the past was due to the training providers having inadequate and inaccurate information about training needs. How can an architect custom-design a house to match your requirements without an exact description of what you want and need? It is the responsibility of the SO team to write or obtain a precise SOW based on solid performance analysis and training needs assessment, then to find the training experts to design the program (or select an “off-the-shelf” one). Don’t expect the training provider to produce results without knowing the trainee’s capacity, needs and objectives. Design Cost-Effective Programs Correlations have been found between training impact and the trainee’s involvement in and endorsement of the training program. In other words, trainees must buy into their program, agree to acquire the skills and knowledge, and commit to applying (and sharing) their new skills and knowledge with other employees at the work place. Prior to the start of the training program, trainees should clearly understand what is expected of them upon their return. Trainees need to know how their training fits into resolving performance problems at the institution and contribute to the SO team’s specific IR. Correlations also exist between training impact and the degree of interest and involvement on the part of the employee’s supervisor in obtaining results from the investment in training. Supervisors need to buy into the training program in order to smooth the way for application of changes brought back by the trainees. For training that targets attitude changes, selection criteria might include leadership qualities, aptitude, a strong desire for achievement, and a commitment to the apply the fruits of the training program to the organization. For larger, in-country skill-building programs targeting organizational units (for instance, in accounting, computers, marketing, etc.), the SO team may not require the same leadership potential but may focus on skill aptitudes. In addition to content questions, the SO teams should consider other factors in selecting a training provider, such as location (in-country, third-country, U.S.), format of training (large conference, adult training-style workshop, intensive course, degree studies, managerial, vocational) and duration (short￾term, long-term). In preparing budgets, be sure to include planning, assessing and monitoring services as needed, as well as the critical follow-on activities to ensure that skills and knowledge acquired are applied at the work place. USAID/G/HCD: Quick Reference to Results-Oriented Training Page 6 Q: It looks like you’re trying to give a new name to an old concept – follow-on. Realistically, how can SO teams or HCD officers help trainees apply their new skills – what you call Knowledge, Skills and Attitudes – when our team has to move on to new training? This is the real problem. What actually works in follow-on? A: First, you’ve recognized that you must do something, and that’s a good start. If you do nothing, you will not know what results your team’s investment has yielded. Second, you’ve identified the problem - lack of knowledge about what works. Follow-on takes many forms, some as simple as a telephone call or on-site visit once every quarter. Others may require a mission SOW to obtain services from a contractor to help the team and its partners leverage the team’s investment in training to ensure results. Support the Application of KSA After Training Transferring the benefits of training to the work place is key to achieving the results your team is anticipating. The best way to ensure adequate and appropriate post-training support is to plan it early-on when designing the program with the team’s organizational partners and the trainees. Make a simple plan, with benchmarks, of what the trainees will do upon return, and then obtain agreement on the details. This becomes the team’s training contract with the organization and the individual trainees, and almost by itself, will do more to promote application of KSA than any amount of pressure to do “follow-up.”. These are some of the types of support the team may need to undertake to ensure that the trainees do not revert to old ways: • maintain strong links (site-visits, phone calls, meetings, etc.) with supervisors and partners with whom the team collaborated while designing the training program • keep communication lines open directly with trainees to see whether (and how) they are applying their new knowledge; stay close to their success or failures; • reinforce success stories among trainees (from different organizations or within the same) to help bolster their attempts to introduce work-place improvements; • involve USAID leadership in monitoring and pressuring the partner institution to accept and welcome change from their employees • lean on returned USAID trainees (through alumni or professional groups) to assist the team in helping employees apply their skills and bring about change; These are just a few ways a team and its partners can support the all-important transfer to the work place of the new KSA that USAID has provided. If the SO team does not have the time or means to provide post-training support, then contract it out to one of the following: a) a mission institutional contractor already working with the partner institution on another activity; b) the training provider that organizes the training, if in-country; or c) a discrete “follow-on” activity described in a SOW that could USAID/G/HCD: Quick Reference to Results-Oriented Training Page 7 Q: What is the easiest and least-costly way to monitor results? A: Good question. Two keys to success: plan monitoring early and keep it simple. Collect only the data you absolutely need to track changes in your IR indicators. Remember that if you have applied these Best Practices from the beginning, you will already have most, if not all, the indicators and information you need (refer to the Performance Gap Analysis and TNA earlier). Armed with your indicators and knowledge about what performance changes the team is hoping for, employ Kirkpatrick’s four levels of evaluation (described below) as a guide. It’s simple, time-tested and easy to understand. Monitoring is not something a consultant does after the training is over. We Monitor Training for Results be implemented through GTD. Because a team may not have the capacity to manage post-training support is no excuse for not doing it! In the past, USAID evaluated training by measuring numbers of participants and training programs and the resources spent. The focus was on keeping the participant numbers high. For decades missions aimed at creating a “critical mass” of trained people, often to fill public-sector positions needed by growing economies or newly-independent nations. Evaluations were generally “tracer studies” identifying the percentage of participants who returned and to what extent they occupied positions of higher authority. Few ever investigated carefully the changes that resulted from these investments. In the reengineered environment, SO teams need to stimulate institutional change that can lead to progress in reaching their IRs. In this environment, the focus shifts from training individuals to helping institutions improve performance. How can this process be measured ? In a landmark work published in the late 1950s for U.S. corporations, Donald Kirkpatrick proposed to aim training evaluations at four levels of inquiry, described in the box below: USAID/G/HCD: Quick Reference to Results-Oriented Training Page 8 Kirkpatrick’s Four Levels of Evaluation Level 1 Reaction: how did participants react to the training (content, instructors, materials, location, etc.); usually conducted at the end of the program; Level 2 Learning: what knowledge, skills or attitudes did the trainees actually acquire? Training providers normally are asked to demonstrate through “before and after” testing that they have effectively transferred the KSA requested; Level 3 Application: have the participants applied the newly acquired KSA on-the-job? The research to answer this question is usually conducted from 3 to 6 months after return. Level 4 Organizational Performance Change: discovers whether the application of KSA noted in Level 3 (if any) produced measurable changes in terms of the quantity and quality of the organization’s output or production; often called simply “results,” level 4 is the highest evaluation finding. In a sense, monitoring can be viewed as the reverse of planning: one plans “down” from the sector, to the SO, the IR, the institution and finally, to the work unit; one then collects performance data in order to monitor results from training “up” from the work unit to the institution and back to the IRs. SO teams should try to attribute changes that are measured to the application of the KSAs the transfer of which USAID-funded training supported. It is not sufficient to simply note that an institution is improving. Monitoring (and the application of all the Best Practices) will help SO teams understand the impact of their training investments, which in turn adds knowledge to USAID as a learning organization. If proper monitoring is conducted by SO teams throughout the year, the burden of gathering reporting data at the last minute, such as “R-4 time,” will be largely avoided. USAID/G/HCD: Quick Reference to Results-Oriented Training Page 9 REFERENCES AMEX International, Inc. and Creative Associates, Inc. 1996. Best Practices Guide: A Road Map to Results Oriented Training. HRDA Project, USAID. Washington, D.C. AMEX International, Inc. 1997. The Principles and Application of Best Practices for Strategic Training: A Resource Manual for Strategic Objective Teams and Activity Managers. HRDA Project, USAID. Washington, D.C. Brinkerhoff, Robert. 1987. Achieving Results from Training. Jossey-Bass Inc., San Francisco, California. Brinkerhoff, Robert, and Stephen Gill. 1994. The Learning Alliance. Jossey-Bass Inc., San Francisco, California. Broad, Mary and John Newstrom. 1992. Transfer of Training: Action Packed Strategies to Ensure High Payoff from Training Investments. Addison-Wesley, Reading, Massachusetts. Kirkpatrick, Donald. 1994. Evaluating Training Programs, the Four Levels. Barrett-Koehler Publishers, Inc., San Francisco, California. Otero, Cecilia. 1997. Training as a Development Tool. Academy for Educational Development. USAID. Washington, D.C. Robinson, Dana Gaines and James C. Robinson. 1995. Performance Consulting: Moving Beyond Training. Berrett Koehler Publishers, Inc., San Francisco, California. Robinson, Dana Gaines and James C. Robinson. 1989. Training for Impact: How to Link Training to Business Needs and Measure the Results. Jossey-Bass Inc., San Francisco, California. USAID/G/HCD: Quick Reference to Results-Oriented Training Page 10 BEST PRACTICES FOR RESULTS-ORIENTED TRAINING (From HRDA Best Practices Guide, 1996) ‘ Contribute to Strategic Planning ‘ Collaborate with Stakeholders ‘ Identify Training Needs in Partner Institutions ‘ Contribute to Improvements in Organizational Performance ‘ Select Trainees with the Greatest Potential to Initiate and Support Change ‘ Work with Trainees to focus on Performance Improvements ‘ Design Cost-Effective and Targeted Training Programs ‘ Monitor Training for Results ‘ Provide Follow-On support to Trainees and Partner Institutions The Best Practices Guide, from which the above is drawn, is a “road map to help navigate the process of training.” The destination point is where training makes its optimal contribution to achieving USAID strategic objectives. The practices are road signs of what needs to be done to achieve results. The practices are not listed in sequential order; you may implement elements of a number of practices at various points in the process of managing training for results. Page 54 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Annex D USAID/Egypt Options Meeting Powerpoint Presentation Note: This Annex consists of a) the information handed out prior to the presentation made to senior USAID staff on Feburary 24, 2001, and b) the Presentation slides reformatted for Microsoft Word for ease of including in this electronic version of the report. 55 Options Meeting on USAID Management of Training DT2 Evaluation Team February 25, 2001 2:00 – 3:00 pm Agenda I. Background and Update II. Findings of the Evaluation Team III. Options and Recommendations for the Future 20 min. IV. Questions and Discussion 15 min. V. Thoughts on Long-Term Academic Training 10 min. VI. Additional Questions and Summary 15 min. Total 60 min. DT2 Evaluation Team Members Andrew Gilboy Barbara Hunt Jeanne Moulton Gerald Wein Aguirre International Background In 1995, USAID/Egypt approved the Development Training 2 Project and in 1997 signed a six￾year contract (three core years plus three option years) with the Institute for International Education (IIE) that centralized most of the mission’s training portfolio. DT2 was to plan, procure, and manage all participant training (U.S. and third-country training) and to assist in the design and management of in-country training as requested by Strategic Objective Teams. Until then, the large number of participant training programs managed by technical contractors were not consistently following agency policies, federal laws, or best practices, and thus expos￾ing the mission to financial, legal, and programmatic vulnerabilities. Training was often even an afterthought as far as strategic and project planning were concerned. This had been a matter of increasing concern since the early 1990s when audit findings led to a decision to centralize the orientation, follow-on, and evaluation components of participant training. The DT2 Project ex￾panded centralization to include the full range of design and management concerns of re￾engineered training. Evaluation Objectives Develop options, based on an evaluation of DT2, for the management of training, participant and in-country, after the end of DT2 in 2003 and conduct a full-mission Training Options Meeting to consider evaluation findings, options on training management issues, and strategies to improve Strategic Objective Teams’ participatory role in integrating training into results package activi￾ties. Page 56 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt USAID-Egypt Management of Training Assessment of “DT2” Options for the Future Aguirre International, Inc. February 2001 Purpose of the “Options” Meeting Purposes of the DT2 Evaluation Evaluate the effectiveness of the DT2 centralized management mechanism Develop options and recommendations for future training Purposes of the meeting Present findings and options concerning the management of training under DT2 Assist USAID in finding common ground on key issues that will determine the shape of training in Egypt after 2002 57 The “Options Meeting” Agenda What did DT2 Set Out to Do? 1. Correct non-compliance in implementation of overseas participant training pre-departure briefings medical exams, insurance, visas, etc. monitoring during training ...What did DT2 Set Out to Do? 2. Streamline training management Centralize funding to ease Mission burden Reduce Mission Training Office Increase use of Best Practices for results-oriented training ...What did DT2 Set Out to Do? 3. Ensure that training contributed to Mission SO/RP Results Linking Reengineering and Best Practices Training based on needs assessments of target institutions Participants selected by organizational need Training design / specifications linked to desired change Work place follow-up ...What did DT2 Set Out to Do? Key Assumption The level of USAID reengineering would be sufficient to enable the SO/RP teams to determine training needs in consort with their TA and Egyptian partners Page 58 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt DT2 - What happened? The Early Period of DT2 Management (1997- 98) SO/RP Teams did not take the lead in assessing training needs sufficiently DT2 took seriously its mandate to apply Best Practices Emphasis on “getting the process right” before embarking on training. Over￾ambitious TNAs impeded implementation TNA and other processes tarnished DT2-USAID-TA Con-tractor relationship and slowed pipeline Confusion about where In-Country Training is managed DT2 - What happened? The Current Period of DT2 Management (1999 - 2001) DT2 responded to criticisms in the Jan ‘99 evaluation Training numbers increased dramatically DT2 became more responsive to clients Pipeline unblocked Management emphasized facilitating implementation over imposing Best Practices processes 59 Findings Process Determining Training Needs TA Contractors take the lead Inconsistent quality Few rapid appraisal methods being used Large TNAs did not yield more results Absence of TNAs lead to unneeded training … More Findings Training Design and Specifications TA Contractors taking the lead generally works well Exception: process of establishing specs DT2 not always viewed as “customer friendly” Procurement DT2 generally provides useful and valued service Exception: tailored In-Country Training … More Findings Processing Overseas Participants DT2 generally provides efficient service Some TA Contractors do not understand USAID requirements USAID/Egypt participant training regulations too rigid Monitoring, Follow-on & Evaluation DT2 has strengthened practices TA Contractors do not always value or want some of these services … More Findings Page 60 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Centralized Funding Diminished sense of ownership among users Reduced cost consciousness Separated decisions on training from decisions on TA Were DT2 Objectives Met? 1. Is USAID in compliance? Problem largely resolved, but some management issues remain 2. Were processes streamlined? Many improvements made, but concerns remain 3. Did DT2-managed training contribute to SO/RP results? Not much. The emphasis on process limited DT2’s effectiveness in introducing results-oriented training Centralized funding distorted decision-making Funding Options Centralized Maintain the present system Mission Management How much money for training How should we allocate training resources among SO/RP Teams RP/Teams & TA Contractors How should we spend the training $ and the unearmarked funds 61 Decentralized Change the system Mission Management How much money for each SO? RP/Teams & TA Contractors How much for training and how much for other activities? Page 62 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Centralized vs. Decentralized Funding Funding Recommendation Decentralize Funding to the SO Teams Advantages: Integrate TA and Training Uses resources efficiently Places resource decisions in hands of those responsible for results Improves budgeting and planning Enhances flexibility Management Options Which organization manages each functions? RP Team/TA Contractor or DT3? Functions • Needs assessment and training plan • Annual plans and budgets • Training design (course specifications) • Procurement • Processing of overseas training • Monitoring, follow-on and evaluation Management Recommendations Functions and responsibilities Lead Organization RP/TAC DT2 • Needs assessment and training plan X • Annual plans and budgets X Tailored in-country Off-the-shelf in-country Overseas short-term Overseas academic • Training Design and Specs X 63 Management Recommendations Functions and responsibilities Lead Organization RP/TAC DT2 • Procurement Tailored in-country X X Off-the-shelf in-country X X Overseas short-term X X Overseas long-term academic X • Processing of overseas participants X • Monitoring, follow-on & evaluation X X Management Recommendations Summary of Principal DT3 Services 1. Procure off-the-shelf training 2. Manage the process for overseas training 3. Advise and assist SO/RP Teams, TA Contractors and Egyptian training providers as re￾quested on … Best Practices and Performance Improvement Strategies 4. Help strengthen the capacity of Egyptian training providers Management Recommendations New Directions For DT3: • Shift from emphasis on process enforcement to one of advising and assisting in Best Practices • Train and/or recruit staff with requisite skills and experience to advise in Best Practices and im￾prove customer service • Introduce a more flexible staffing system in order to adjust LOE to USAID’s evolving needs for training management Page 64 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Management Recommendations New Directions For USAID: • Revise its oversight role to reduce the number of approval actions and to eliminate its role in re￾viewing training budgets • Increase its role in promoting Best Practices for training • Consider using the new START contract for DT3 • Use the DT3 Performance Improvement Advisory Services to increase USAID staff understand￾ing of results-oriented training • Revise regulations to be in line with Mission objectives # # # # # # # # # # # # # 65 Note: The topic presented below was not part of the Scope of Work but surfaced as an issue for which the Mission requested guidance. The Team therefore included these points in its presentation to USAID officials. Graduate Training for Egyptians at U.S. Universities: Framing the Discussion • Prior to committing to program, determine reasons for the relatively small number of Egyptians at U.S. universities (cost, qualifications, culture, other opportunities?) • Build on the Lessons Learned from USAID’s extensive experience in large-scale degree programs world-wide Fields of study to be sponsored must be based on market demand for skills General “scholarship” programs for “Best and Brightest” can lead to high non-return rates and low returned participant employment Training “able” individuals unassociated with institutions ignores Best Practices and reduces likeli￾hood of impact Framing the Discussion • Focus on Development – Technical areas linked to USAID’s Strategic Objectives – Train critical mass within targeted institutions or industries – Consider institutional twinning programs with graduate degree programs and exchanges Framing the Discussion • Selection & Preparation – Invest up front in recruitment, establishing criteria, building solid selection procedures – Prepare finalists for required entrance exams (GMAT, etc.) – Select the BEST candidates regardless of English level – Offer sufficient in-country and U.S. ELT – Consider introducing quotas for candidates from target populations: women, low-income, rural. Page 66 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Framing the Discussion • Carefully monitor participants in while training – Increases return rate and results – Helps fine-tune program to market demand – Reduces cultural impediments to success # # # # # # # # # # 67 Annex E Scope of Work and Aguirre Team Work Plan Page 68 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt SCOPE OF WORK DEVELOPMENT TRAINING 2 (DT2) PROJECT EVALUATION AND OPTIONS DEVELOPMENT I. OBJECTIVE USAID/Egypt, Office of Human Development and Democracy/ Education and Training (HDD/ET), requests a proposal from Aguirre International through a Task Order to the Global Evaluation and Monitoring (GEM) IQC contract to evaluate and assess major design, management, and strategy issues of the three component parts of the Development Training 2 Project (DT2) and to recommend options for the management of the mission’s training portfolio after the end of DT2 in 2003. The audience for this evaluation will be mission senior management, Strategic Objective Teams, and HDD/ET staff and contractors. The purposes of this evaluation and options development activity are to: • Develop options, based on an evaluation of DT2, for the management of training, participant and in￾country, after the end of DT2 in 2003 and conduct a full-mission Training Options Meeting to consider evaluation findings, options on training management issues, and strategies to improve Strategic Objective Teams’ participatory role in integrating training into results package activities. • Evaluate the effectiveness of two English language activities placed under DT2, Integrated English Language Program II (IELP-II) and the English Language Testing and Training Program (ELTT), in 1) achieving project objectives and sustainability by designing and implementing systems which will insure continued preparation of providers after projects are completed and 2) providing lessons and models that could be applied to future USAID-funded programs related to English language teaching, teacher training, strengthening of public and private training providers and overall human capacity development in Egypt. This activity will be conducted in two stages: Stage One: DT2 Components 2 and 3, Integrated English Language Program II (IELP-II), and Eng￾lish Language Testing and Training (ELTT), beginning o/a January 3, 2001, for ap￾proximately four weeks. Stage Two: DT2 Component 1, Training, beginning o/a January 27, 2001, for approximately five weeks. BACKGROUND In 1995, USAID/Egypt approved the Development Training 2 Project and in 1997 signed a six-year contract (three core years plus three option years) with the Institute for International Education (IIE) that centralized most of the mission’s training portfolio. DT2 was to plan, procure, and manage all participant training (U.S. and third-country training) and to assist in the design and management of in-country training as requested by Strategic Objective Teams. 69 Until then, the large number of participant training programs managed by technical contractors were not consistently following agency policies, federal laws, or best practices, and thus exposing the mission to financial, legal, and programmatic vulnerabilities. Training was often even an afterthought as far as strategic and project planning were concerned. This had been a matter of increasing concern since the early 1990s when audit findings led to a decision to centralize the orientation, follow-on, and evaluation components of participant training. The DT2 Project expanded centralization to include the full range of design and management concerns of re-engineered training. The purpose of DT2 was to centralize participant training in one mechanism in order 1) to assume greater control and assure adherence to training policy, 2) to provide training expertise to complement technical expertise of technical contractors who usually are not training specialists, thus improving the effectiveness of training investments, 3) to apply the experience of DT1, the findings of mission and agency evaluations and audits of training, and advances in the discipline of training, and 4) to bring training operations and portfolio in line with re-engineering principles and better serve the mission’s Strategic Objectives and results management mandates. The premise of DT2 was that added rigor needed to be brought to bear on the often chaotic implementa￾tion of training, especially in steering the goal away from often scattered transfer of knowledge, skills, and attitudes (KSAs) toward a more effective harnessing of technology transfer to the attainment of ar￾ticulated performance improvements by individuals and groups critical to the attainment of intermediate results. This requires a more holistic approach advocated by training and performance improvement spe￾cialists around the world: to assess the barriers to performance and to attack those amenable to training interventions in coordination with other interventions. DT2 stands as the leading USAID attempt to bring sufficient program critical mass and funding to changing the human behavior contribution to development. Centralization of training in a mechanism designed to practice this approach was only a means, not an end. Ideally, this should be the ordinary way of doing things at the SO activity level. But the perception that it is “more work” to focus on perform￾ance rather than learning and to more closely integrate training and non-training interventions has usually doomed attempts to adopt this approach in other missions. Implementation in USAID/Egypt was both marked by successes and hampered by circumstances which led to management and programmatic conse￾quences that require critical assessment. Chief among the needs of this evaluation with regard to training is arriving at a balance sheet of pros and cons, advantages and disadvantages, of the DT2 approach as it has worked in practice, both to assess mid￾course corrections currently underway and to begin constructing options for post-DT2. This evaluation will take place just after the completion of four years of IIE’s six-year contract, two-thirds of the way through the DT2 effort. In addition to its major component, training management, the umbrella DT2 mechanism also includes two other activities: Integrated English Language Program II (IELP-II). This four- to six-year technical assistance and training program administered by the Academy for Educational Development (AED) and its subcontractor, AMIDEAST, was put under DT2 as a contracting convenience. It will not be included under a new DT2. The goal of IELP-II is to increase the number of qualified English-language providers in the public and export-oriented private sector in Egypt. The program provides technical assistance, educational materials and other resources, and strategic training in Egypt and the U.S. Page 70 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt The framework of IELP-2 activities includes the principal objectives of "an increased number of qualified English language providers. Its three principal results are a) increased number of qualified future and current English language teachers; b) increased number of qualified teachers of English for Specific Purposes (ESP); and c) increased number of qualified providers of English for Occupational Purposes (EOP). The program is designed to strengthen Egypt’s competitive workforce position in the global economy of the 21st century. Of its $53 million budget, $10.4 million in training costs are covered by DT2 pass-through funds. English Language Training and Teaching Program (ELTT). This is a support service for DT2, adminis￾tered by the American University in Cairo (AUC), that enables the mission to meet the U.S. Government requirement that all participants be tested for and meet minimum English language proficiency and un￾dergo cultural orientation for training in the U.S., outlined in USAID’s Automated Directives System (ADS Chapter 253.) It was absorbed into DT2 as part of the centralized training system. ELTT provides both testing and training services. To assess the English proficiency of candidates for technical and aca￾demic training, ELTT uses a battery of tests developed by the American Language Institute/Georgetown University (ALIGU), namely the English Proficiency Test (ALIGU/EPT) for academic participants and the Communicative English Proficiency Assessment (CEPA) for short-term technical programs. ELTT training services are in the form of general English Language Training (ELT) for participants nominated for U.S. training and English for Specific Purposes (ESP) for counterpart staff not nominated for U.S. training but targeted for English improvement. ELTT supports the mission’s SOs by enabling participant training programs in the U.S. to take place. 71 II. STATEMENT OF WORK: STAGE ONE - IELP-II AND ELTTP COMPONENTS The Contractor shall undertake all of the following tasks to fulfill the objectives of this Task Order. After arrival in country, the Contractor shall coordinate with USAID to set up meeting schedules, agendas, and advance information requirements. The outcomes of this coordination and planning shall be incorporated into the Contractor’s Work Plan. A. Background Document Review and Work Plan Development Prior to arrival in country, HDD/ET will forward to the contractor by courier the following documents and other information suggested by USAID/Egypt to gain an understanding of the mission’s strategy and training portfolio. In coordination with USAID/Egypt, the Contractor will develop and submit a Work Plan by the beginning of Week 2 for review and approval, which specifies how, when, and by whom each of the tasks in this SOW shall be accomplished. Necessary background documents to be reviewed in￾clude, but are not necessarily limited to, the following: 1. Background for Stages One and Two USAID/Egypt, Advancing the Partnership: Strategic Plan, FY 2000-2009 USAID/Egypt, R2 for 2002 (Results for FY 1999) USAID/Egypt, OFE [Orientation, Follow-on, Evaluation] Action Memo – Discussion Draft (1993) USAID/Egypt, Action Memo: Centralized Participant Training Services (1994) USAID/RIG/Audit, Audit of USAID/Egypt’s Management of Project-Related Participant Training (1994) USAID/Egypt, Development Training 2 Project Paper (1995) USAID/Egypt, IIE DT2 Contract (1997) 2. Specific for Stage One USAID/Egypt, AED IELP-II Contract (1997) and Modification (2000) Andrew Gilboy, Assessment of the English for Occupational Purposes Component of IELP-II (2000) AED, IELP-II Workplans AED, IELP-II Performance Monitoring Reports (quarterly) AED IELP-2 Sustainability Plan, 1998 and 1999 and 2000 AED IELP-2 Proposal (since the document is very thick it will not be sent to evaluator) USAID/Egypt, AUC ELTTP Contract (1997) Page 72 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Georgetown University, Suzanne Peppin and Leslie Palmer, Technical Assistance Report [Evaluation of English Language Activities under DT1] (1992) Suzanne Peppin and Leslie Palmer, Technical Assistance Report [Evaluation of ELTTP] (1998) and AUC’s report on implementation (2000) AUC, ELTTP Performance Monitoring Reports (quarterly and annually) 3. Specific for Stage Two IIE, Staffing Tree IIE, DT2 Workplans IIE, DT2 Performance Monitoring Reports (Quarterly) IIE, DT2 Step by Step: A User’s Guide (1999) IIE, DT2 Briefing Book (1999) USAID/Egypt, Mission Training Order (1998 version in DT2 User’s Guide; also 2000 draft revisions) HERNS Project, Joyce Kaiser and John Braley, Process Evaluation of USAID/Egypt’s Development Training 2, and IIE Comments (1999) B. Evaluation of IELP-II 1. Issues and Questions to be Addressed a. Overall Examine the effectiveness and quality of the design, implementation and monitoring of key activities: Are IELP-II activities well designed and implemented to meet program objectives? Do they reach the planned target audiences? What are its sustainable features? Determine whether steps taken to date can be expected to contribute to sustainability. How well is IELP-II addressing both the private and public sector foci of the mission’s new strategy? How well is it addressing the geographic focus of the mission’s new strategy? What challenges and opportunities have been faced with key audiences. What constraints or shortcomings are there in the Ministry of Education that create obstacles to sustainability and replicability? How adequately and clearly have objectives been defined in the milestones of the contract? What has been the overall impact in the project’s component areas of pre-service, in-service, testing, English for Specific Purposes (ESP) , English for Occupational Purposes (EOP), and Professional Enhancement Ac- 73 tivities, measured in terms of both the milestones and the broader programs delivered beyond meeting milestones which contribute to the achievement of the objectives of the project? What are the lessons that could be learned/models that can be adapted for future projects? Identify strate￾gies, approaches, processes, and activities that have been particularly successful and could have larger applications to USAID future HCD efforts? This needs to be identified in all six components of the pro￾gram. b. In-Service Training How well is IELP-II creating a basis for sustainability of the model and improving institutional capacity within GDIST (the in-service training department of the Ministry of Education) to design, deliver and manage future training? Will GDIST be able to take over the activities and expand them? What will it take to bring this about? What will it take to enable GDIST to undertake the upgrading of the 500 mid￾level administrators by means of the IELP-II model, instead of massive U.S. training? What efforts have been made to build individual KSA and expertise to provide MOE with qualified staff to continue teacher and supervisor training programs? How well are the GDIST satellite In-Service Training Centers being strengthened? How well are training activities being transitioned and supported by IELP-II to GDIST and its satellite centers? What are the challenges facing the satellites in taking over training of supervisors and other IELP-II activities in the future? What is the level of motivation of the satellites to take over more training responsibilities? How will GDIST be able to continue the capacity building of the satellites? c. Pre- Service Training How well are the selected university Faculty of Education (FOE) Centers of Excellence being strength￾ened? How valid is the current approach to the technical assistance to the Centers in terms of capacity building, serving as resources to other institutions, and impact and sustainability? What efforts have been made to improve institutional capacity at these centers to design, deliver and manage training in the fu￾ture? What efforts have been made to build KSA and expertise to provide FOEs with qualified staff to provide communicative English language and effective interactive teaching skills to future teachers? To what extent are the other Faculty of Education target activities of IELP-II demonstrating impact? How well are they being institutionalized? How might the model of the Centers of Excellence be transferable to other academic specialties? d. Testing Reform How well is the capacity of the National Center for Examinations and Educational Evaluation (NCEEE) being strengthened? What effect will the low number of individuals involved in the test reform activities have on sustainability and the likelihood that the Ministry and the NCEEE will buy into the reforms? What kind of challenges are they facing and how can they be surmounted? e. English for Specific Purposes (ESP) What efforts have been made to improve institutional capacity at selected ESP centers to design, deliver and manage future training? What efforts have been made to build individual KSA and expertise to pro- Page 74 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt vide ESP centers with qualified staff to quality targeted ESP courses to both internal (university EAP) clients and to external (private EOP) clients in the future? f. English for Occupational Purposes (EOP) At the time of the writing of this SOW, there is an ongoing assessment of the EOP component of IELP-II by Andrew Gilboy. That study should be appropriately considered in the evaluation of EOP. What is the prognosis for this activity, given the slow pace thus far? Have original strategies been sub￾stantially altered? Have new programs and strategies been initiated? Have proposed programs or strate￾gies been altered or eliminated? Have the new programs added value? If there are significant changes, has there been a rationale for them? Have the changes contributed to program objectives? How useful might the model of offering technical assistance and capacity building of private sector train￾ers for public and private sector targets be in other activities, such as in school-to-work and workforce development? To what extent could a future IELP-III expand this private sector model? g. Professional Enhancement Evaluate specific activities components based on targeted audience, quality of interventions and appropri￾ateness of resources provided. 2. Recommendations Make recommendations based on meetings with partners, beneficiaries, and implementers. C. Evaluation of ELTT Performance of the ELTT contractor is monitored through the submission of performance reports summa￾rizing progress of the major activities in process in relation to the requirements of the contract, indicating any problems encountered, and proposing remedial actions as appropriate and through frequent personal meetings and telephone conversations. In February 1998, an early process evaluation of the ELTT was done by a two-person team from ALIGU to review the operations of the new contractor. It included on￾site visits and observations of the training and test sessions, assessment of the implementation of the ELTT program for conformance with the SOW and the overall effectiveness and efficiency of the training and testing, and development of recommendations for correcting problems and for making future im￾provements. The purpose of the ELTT component of this evaluation is to: • examine the overall performance and results and judge the effectiveness of the program on customers, and • identify strengths and weaknesses of the ELTT program from assessment of its impact and provide operational recommendations and adjustments to remain supportive of mission strategy and accom￾modate changing mission demands. 75 1. Issues and Questions to be Addressed a. Overall How well does ELTTP meet the needs of the program for timely and effective preparation of individuals for training? How well are the needs of different categories of customers being met? Should ELTT services continue to be centralized? Should they be placed directly under a DT2-type con￾tract rather than remain a separate procurement? Should the mission make more use of the new flexibility in ADS 253 to waive testing and training to ex￾pedite clearance for training of candidates SO Teams feel strongly about and make the SO Teams ac￾countable? Should the mission expect the size of the program to decrease as a result? What resources exist that can assist the COTR in monitoring in-country English language programs? b. English Language Testing What is the value of the ALIGU and CEPA tests? Should ALIGU continue to be given? Are ALIGU Call Forward score requirements set at an appropriate level? Is CEPA a good test of the needs of U.S. short-term participants who will not be in a class setting? Should there be different passing levels of CEPA depending on the type of training? What can be done about candidates who fail CEPA and are not nominated for ELT but repeatedly retake CEPA and is this related to the appropriateness of CEPA? Would it help candidates who rise to the mid￾level course (O2-B) to shift to a special auxiliary course focused on passing CEPA? To what extent has the English testing barrier been part of DT2’s earlier problem of not meeting training numbers? Should CEPA be dropped altogether in light of the more flexible ADS 253 policy on language testing for short￾term programs and more reliance placed on subjective determination of language capacity demonstrated through interviews? c. Nomination Procedures How can nomination procedures and deadlines be better adhered to? How can selection become more accountable and lead to more motivated trainees and supportive supervisors? To what extent are candi￾dates unaware of why they were nominated for ELT and how much does this negatively affect motiva￾tion? To what extent have the auxiliary courses for counterpart individuals become a kind of reward? d. Planning What are the reasons for the difficulty in finalizing English Language Training plans? Is the current level of planning, or lack thereof in some projects, a constraint that will not change and that must be dealt with for the foreseeable future? What is the likely impact on program levels of the waiver flexibility? e. Program Flexibility Is the menu of services broad and flexible enough to meet customer needs? How can the program better accommodate special requests for some customers (on-site training, special scheduling) without compro￾mising quality for others or unduly reducing administrative efficiency? Should there be changes in the Page 76 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt contract to accommodate greater flexibility? Is there a way to amend the contract to raise the total ceiling to accommodate greater demand from SO Teams and partners? What should the balance be between meeting the needs of participants (for ELT) and counterparts (for auxiliary courses/ESP)? Should the distinction be maintained between the two, or should there be one set of courses for participants and counterparts, such as under the previous AUC contract? Should program ceilings be adjusted upward to reflect higher than anticipated demand from counterparts? Are the in￾creased nominations for auxiliary/ESP in part due to the attraction of greater scheduling flexibility? How important should the auxiliary/ESP program be? f. Supervisory Support How can supervisors be induced to better support the special needs of their participants in the program? How can commitment on both sides, participants and supervisors, be strengthened? Would payment of a nominal tuition by participants and counterparts strengthen commitment? Would that present an unneces￾sary accounting and management burden? What is the effect of greatly varying levels of sponsor support (transportation subsidies, flexibility of work schedules) on morale and commitment? g. Training Curriculum Is the curriculum adequate to attain the required proficiency within a reasonable time? Is the length of the program appropriate for most participants who are working and have families? Would fewer hours be suf￾ficient to reach the desired level? How appropriate is the curriculum for Egyptians? 2. Recommendations Make recommendations based on meetings with partners, beneficiaries, and implementers. IV. STATEMENT OF WORK: STAGE TWO - TRAINING COMPONENT Currently concluding the fourth year of its six-year contract, the training component of DT2 had a “proc￾ess evaluation” at the end of the second year. That evaluation assessed how well the contractor was meeting program objectives, analyzed the DT2 contract, examined training costs and compliance with ADS 253 regulations, and made recommendations. The present evaluation will serve in part as a mid￾term assessment but more importantly will address the lessons learned from DT2 as they relate to major decisions the mission has to make regarding the management of training after 2003. That is to say, this evaluation is focused on drawing lessons for the future planning and issues to be examined are chosen with that end in mind. This exercise is undertaken at this time because of the need to address manage￾ment issues of concern to SO Teams and because of the upcoming departures of both the COTR and the HDD/ET Office Director in 2001. 77 1. Issues and Questions to be Addressed a. Centralization of Training What have been the gains and losses and lessons learned from the centralization of participant training? Has the centralized approach resulted in more consciously integrating training with other interventions, in using the performance improvement approach, and in more closely fol￾lowing agency training policies, i.e. in using training best practices? To what extent has centralization helped promote best practices within SO Teams and their tech￾nical contractors? Do any training programs currently implemented by SO Teams and contrac￾tors serve as examples of this? What would be gained or lost from decentralizing training? To what extent is centralization in keeping with re-engineering? In what ways is it not? DT2-Specific Approaches A major aspect of the DT2 approach was the Training Needs Assessments. How well were the TNAs done? Did the magnitude of the TNAs overwhelm other aspect of the DT2 approach? How effective is training based on TNAs vs. programs not based on them? Was the impact of training based on TNAs worth the investment in TNAs and other associated costs? In light of the need to update them? To what extent did the level of success of DT2 among various SO Teams depend on their level of re￾engineering? To what extent would that play a role in future training management strategies? Can any conclusions be made regarding the impact of DT2’s approach to training, the training delivered, and its impact on target organizations and individuals, on SO results? What aspects of DT2 most lend themselves to promoting the stronger integration of training? b. Funding What was the effect of the high level of funding and the way it was allocated on the working relationship between IIE and SO Teams and technical assistance contractors? On the attitude and cooperation toward training and DT2 among SO teams and contractors? Has USAID and contractor management of DT2 resulted in SO Teams competing for funding resources? What can be said about the impact of the 1999 “scrubbing” exercise and the kind and levels of training that survived? What are the pros and cons of allocating funds directly to DT2 vs. via SO Teams? How would this work within the mission’s budgeting system? Would SO Teams buy as much training if it were from their own budgets? Is more training being done simply because of the availability of re￾sources? What conclusions can be made about the comparative cost/benefit of training through a cen￾tralized mechanism versus through other options? c. In-country Training Page 78 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Who should manage generic in-country training? Technical? How well did technical office management of all in-country training work in the past? What criteria should be used by SO Teams in deciding whether to incorporate an in-country training pro￾gram in a technical assistance contract or have it managed by a central contractor? What is likely to be the effect on in-country training of returning programming and budgetary control to SO Teams? d. IIE Management Was and is there friction between IIE and SO Teams? If so, what were the reasons for it and how has the current IIE staff dealt with this problem? What role did the level of rigor on the part of IIE play in the receptivity of SO Teams to im￾proved training practices? What were the effects of IIE’s more recent flexibility following the 1999 evaluation? What IIE management innovations have worked? What other management reforms are recommended? What is needed to make the management of centralized training fully workable and acceptable? How well is the centralization of TraiNet under DT2 working? e. Overall Impact How well is the mission’s training portfolio serving women and geographic areas targeted in the new mission strategy? How well is it serving Egyptian private sector needs? What are the constraints in these areas and how should they be dealt with? What features should a new model have to promote new mis￾sion strategic needs? How can the mission best promote stronger integration of training? What options other than a centralized model are there to promote this? Would the model of an individual serving as an ad￾visor within the mission staff be effective in promoting best practices and furthering SO Team ownership? Within the centralized option, which features of DT2 should be retained and which dropped from a DT3? 2. Recommendations: Options for Future Training Management Planning The evaluation team shall plan and facilitate a mission-wide Training Options Meeting to present and lead a discussion of options based on Stage Two evaluation findings. The meeting should be attended by as broad a range of interested and affected individuals as can be induced to attend, prepared to focus on the potential impact of the various options on their SO Teams’ operations. Rapport between the evaluation team and mission staff built up during the evaluation must be utilized to maximize attendance and a seri￾ous discussion of the issues. The evaluation team, with the support of senior mission staff, should cata￾lyze the mission to come to grips with these training management issues. 79 The evaluation team shall determine the organizing principle behind the categorization of the options and the issues that shape them. They are not limited by the categories under which the questions here were organized if a more fruitful way of categorizing is evident at the end of the evaluation. The final report shall set out the evaluation findings, the options presented at the meeting, a summary of the discussion of the options, and concluding recommendations. V. REPORTING REQUIREMENTS AND DELIVERABLES A. Work Plan At the beginning of Week 2 of each visit, the Contractor shall submit five copies of a draft Work Plan to HDD/ET for review and approval. The Work Plan shall specify the Contractor’s approach to completing all tasks identified in the relevant Statement of Work (Section III for Stage One, IV for Stage Two), in￾cluding the specific strategy, plan, and schedule for obtaining required information, carrying out inter￾views or site visits, and producing the required documents. The Contractor shall proceed to work ac￾cording to its Work Plan while awaiting USAID/Egypt comments. The mission shall provide comments to the Contractor within three days of receipt of the Plan. The Contractor shall address all comments re￾ceived in revising and finalizing the Work Plan and shall submit five copies of the final Work Plan within four days after receipt of USAID/Egypt’s comments. B. Evaluation of and Recommendations on IELP-II and ELTTP (Stage One) At the beginning of Week 2, the Contractor shall submit five copies of a draft outline of the evaluation report on IELP-II and ELTT to the mission for review and approval. The report outline shall identify all the sections of the final report. The Contractor shall continue work on the evaluation while awaiting the mission’s comments on the outline. USAID/Egypt shall provide comments on the outline within three days of receipt. The Contractor shall address all comments received in revising and finalizing the report outline and shall submit five copies of the final report outline within two days after receipt of the mission’s comments. At the beginning of Week 4, the Contractor shall submit five copies of the draft evaluation report for re￾view and approval. The evaluation report shall follow the outline approved previously and shall not exceed 30 pages, not including appendices. The mission’s comments should be received within three days and the last two days will be devoted to discussions on requested revisions. The final report shall be submitted within two weeks of the end of the visit and shall be provided in Microsoft Word 97 and five hard copies. C. Evaluation of DT2 and Options for Future Training Management (Stage Two) At the beginning of Week 2, the Contractor shall submit five copies of a draft outline of the evaluation report on the training component of DT2 to USAID/Egypt for review approval. The report outline shall identify all the sections of the final report. The Contractor shall continue work on the evaluation while awaiting the mission’s comments on the outline. The mission shall provide comments on the outline within three days of receipt. The Contractor shall address all comments received in revising and finaliz￾ing the report outline and shall submit five copies of the final report outline within two days after receipt of the mission’s comments. Page 80 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Also at the beginning of Week 2 or before, the Contractor shall confer with HDD/ET staff and, in con￾sultation with senior management, set a date for the mission-wide Options Meeting at which the draft re￾port with training management options will be discussed. The date should be in Week 5. At the end of Week 4, or not less than three days before the date of the Options Meeting, the Contractor shall submit the draft Evaluation and Options Report for review, in sufficient number of copies for all expected to read it and attend. On the basis of written comments on the draft and discussion at the Options Meeting, the Contractor shall submit the final report within two weeks after leaving Egypt. D. Periodic Progress Briefings After arrival in country, the Contractor shall conduct an initial briefing with HDD/ET staff to discuss the plan of action. Afterwards, the Contractor shall provide weekly progress briefings for the office and other mission personnel, as appropriate. The schedule for these briefings shall be agreed upon between the Contractor and HDD/ET during the initial briefing. E. Format of Reports The two final reports should be formatted as follows: • Executive Summary • Table of Contents • Body of the Report • Appendices VI. PERFORMANCE PERIOD The evaluation of the two English language programs should begin o/a January 3, 2001. For Stage One, the in-country work should be completed and a draft report provided to HDD/ET o/a January 28. It is anticipated that the Stage One evaluation team will depart Egypt by January 31, 2001. HDD/ET must receive a final report by February 15, 2001. The Stage Two evaluation team should arrive in Egypt o/a January 27, 2001 and provide a draft report to HDD/ET by o/a February 26. It is anticipated that the Stage Two team will depart Egypt by March 1, 2001. A final report for Stage Two must be received by March 15, 2001. VII. SKILLS REQUIRED The Contractor shall provide two teams of specialists to carry out the tasks described in this SOW. Although the teams may not have the exact breakdown of expertise on an individual basis as indicated below, the Contractor shall assure that the overall composition of the teams includes the following mix of expertise and experience: evaluation; contemporary training approaches, performance improvement, and organizational development design and management; private sector training; English language testing and training; strategic planning; and USAID procedures 81 and ADS regulations. Each member shall have excellent English language speaking and writing skills. The team members should have a broad range of experience in the areas of expertise as noted above as well as extensive experience with training programs in developing countries. VIII. TECHNICAL DIRECTION The two teams will work under the general guidance of HDD/ET Director Sally Patton, with day to day direction from DT2 COTR James Van Den Bos. For Stage One, assistance will be provided by IELP-II COTR Mona Zikry and ELTTP COTR Salwa Nashed. IX. LOGISTICAL SUPPORT The Contractor shall be responsible for providing all necessary logistical support. Logistical support in￾cludes, but is not necessarily limited to, travel arrangements, lodging, international and host country travel, lodging, office and secretarial support, etc. X. USE OF GOVERNMENT FACILITIES AND PERSONNEL The Contractor and any employee or consultant of the Contractor is prohibited from using U.S. Govern￾ment facilities, such as office space and equipment or U.S. Government clerical and technical personnel in the performance of this Task Order, unless authorized. XI. DUTY POST The contractor shall perform work under this Task Order in Egypt and the U.S. ACCESS TO CLASSIFIED INFORMATION The Contractor shall have no access to classified information. XIII. WORKWEEK A six-day workweek is authorized. The normal workweek in Egypt is Sunday through Thursday. EVALUATION OF REPORTS AND DELIVERABLES Sally Patton, Director, Office of Human Development and Democracy/Education and Training, and James Van Den Bos, DT2 COTR, shall evaluate the Work Plan, the outline for and the final reports, and the format and conduct of the Training Options Meeting. Each report and deliverable shall be reviewed and evaluated based on its responsiveness to the Scope of Work. This includes depth of analysis, organization of reports, clarity, consistency, cohesiveness, timeliness, quality of workmanship, adherence to and compliance with ADS requirements, and responsiveness to mission direction and comment. + Page 82 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Development Training 2 Project Evaluation and Options Development Strategy and Work Plan Submitted By: Aguirre International Evaluation Team Andrew C. Gilboy Barbara Hunt Jeanne Moulton Gerald Wein February 11, 2001 I. Strategy The team spent the first week meeting with USAID and IIE officials to explore the questions highlighted in the Scope of Work and obtain additional documentation that will be needed. The following strategy is proposed that will enable the team to arrive at findings concerning the management of USAID-funded training through the DT2 centralized mechanism and to develop options and recommendations for future training that the Mission can con￾sider. A. Methodology The team has developed a methodology in order to guide data-gathering over the two-week period available. The following section explains the rationale behind the methodology selected that will ensure that information obtained is comprehensive and readily available. • A statistically-significant survey of returned participants to determine the level of impact of training on their work units is beyond the SOW and not the primary focus of this "forward-looking" evaluation. How￾ever, we will explore perceptions held by interviewees that training may have had an impact on partner in￾stitutions. We will also look at how impact data is collected, disseminated and used. • An evaluation of contractor performance is not the primary focus of the SOW. However, awareness of DT2's effectiveness is essential to developing future options and recommendations. • Structured interviews are the principal tool that Team members will use to gather information. Some small group interviews or focus groups may be organized, if time and logistics permit, to produce additional per￾spectives which might not otherwise be generated. • By using structured interview formats containing standardized as well as opened-ended questions, the Team will be able to quantify selected responses to supplement the more “anecdotal" information gathered. 83 • Approximately 50 USAID SO and RP staff, TA Contractor staff and partner institutions that have used DT2 will be interviewed by Team members. Approximately half of the interviewees will have been in￾volved in selected Case Studies. The other half will have had some significant relationship with the DT2 Project. • At least 4 Case Studies will be developed to reveal the dynamics of training management in Egypt as seen from within a single completed training event. The case studies will be selected according to criteria the team has established and will involve multiple training "programs". In addition to USAID, IIE, TA con￾tractors and EPIs, ICT training providers and some participants will be interviewed where possible. Gath￾ering information at this level is secondary to the other targets described above. • Develop a balance sheet of “pros and cons” that will help determine the benefits, risks and disadvantages of managing training through a central contractor. The balance sheet will include details about the types of training and management services needed. • The Options Meeting described in the SOW will provide a forum for Mission decision-makers to consider the Team’s findings and recommendations and ask questions about the various options put forward. Be￾cause of time constraints and other factors, the meeting will be limited to 1.5 hours, of which the team’s presentation will be approximately 45 minutes. B. Principal Issues to be Addressed The methodology described above will guide the team in its information-gathering and analysis phase. The major issues highlighted in the Scope of Work, which are repeated below, will be addressed by the Team by applying the methodology. Centralization of Training What have been the gains and losses and lessons learned from the centralization of participant training? Has the centralized approach resulted in more consciously integrating training with other interventions, in using the per￾formance improvement approach, and in more closely following agency training policies, i.e. in using training best practices? To what extent has centralization helped promote best practices within SO Teams and their technical contractors? Do any training programs currently implemented by SO Teams and contractors serve as examples of this? What would be gained or lost from decentralizing training? To what extent is centralization in keeping with re-engineering? In what ways is it not? (The following is added to the SOW) • How should DT3 be designed to be effective and efficient for USAID under various degrees of reengineering? • Regarding centralized training management under the follow-on to DT2, how can/should the Mission ration training? • Regarding centralized training management under the follow-on to DT2, how can/should the Mission ration management intensive training? DT2-Specific Approaches A major aspect of the DT2 approach was the Training Needs Assessments. How well were the TNAs done? Did the magnitude of the TNAs overwhelm other aspect of the DT2 approach? How effective is training based on TNAs Page 84 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt vs. programs not based on them? Was the impact of training based on TNAs worth the investment in TNAs and other associated costs? In light of the need to update them? To what extent did the level of success of DT2 among various SO Teams depend on their level of re-engineering? To what extent would that play a role in future training management strategies? Can any conclusions be made regarding the impact of DT2’s approach to training, the training delivered, and its impact on target organizations and individuals, on SO results? What aspects of DT2 most lend themselves to promoting the stronger integration of training? Funding What was the effect of the high level of funding and the way it was allocated on the working relationship between IIE and SO Teams and technical assistance contractors? On the attitude and cooperation toward training and DT2 among SO teams and contractors? Has USAID and contractor management of DT2 resulted in SO Teams compet￾ing for funding resources? What can be said about the impact of the 1999 “scrubbing” exercise and the kind and levels of training that sur￾vived? What are the pros and cons of allocating funds directly to DT2 vs. via SO Teams? How would this work within the mission’s budgeting system? Would SO Teams buy as much training if it were from their own budgets? Is more training being done simply because of the availability of resources? What conclusions can be made about the comparative cost/benefit of training through a centralized mechanism versus through other options? (The following is added to the SOW) • Regarding contractor management of training, what is the impact of an increase in the demand for procuring highly-specialized training or consulting services? • Are there specific problems when DT2 manages training outside a USAID Results Package? In-Country Training (ICT) Who should manage generic in-country training? Technical? How well did technical office management of all in￾country training work in the past? What criteria should be used by SO Teams in deciding whether to incorporate an in-country training program in a technical assistance contract or have it managed by a central contractor? What is likely to be the effect on in-country training of returning programming and budgetary control to SO Teams? IIE Management Was and is there friction between IIE and SO Teams? If so, what were the reasons for it and how has the current IIE staff dealt with this problem? What role did the level of rigor on the part of IIE play in the receptivity of SO Teams to improved training prac￾tices? What were the effects of IIE’s more recent flexibility following the 1999 evaluation? What IIE management innovations have worked? What other management reforms are recommended? What is needed to make the management of centralized training fully workable and acceptable? 85 How well is the centralization of TraiNet under DT2 working? Overall Impact and Special Targets How well is the mission’s training portfolio serving women and geographic areas targeted in the new mission strat￾egy? How well is it serving Egyptian private sector needs? What are the constraints in these areas and how should they be dealt with? What features should a new model have to promote new mission strategic needs? How can the mission best promote stronger integration of training? What options other than a centralized model are there to promote this? Would the model of an individual serving as an advisor within the mission staff be effective in promoting best practices and furthering SO Team ownership? Within the centralized option, which features of DT2 should be retained and which dropped from a DT3? The team will present its findings and develep options for DT3 at the Options Meeting and in the final report. C. Allocation of Team Resources The Work Plan on the following page specifies the tasks and time frame that the Team expects to adhere during the assignment. The tables on this page illustrate the way in which the team expects to spend its time. Use of the Team’s Resources By Level of Effort Tasks Level of Effort in % Document Review 15 Interviews 55 Analysis 15 Presentation and Report Finalization 15 Total 100 By Interview Population Tasks Level of Effort in % USAID 45 TA Contractors 20 IIE 15 EPIs 10 Training Providers 5 Participants 5 Total 100 II. Work Plan February 11, 2001 Week Tasks 1 February 3 – 10 Discovery & Preparation • Meet with USAID/HDD/ET and IIE to review SOW and agree to the evaluation’s primary focus • Review documents and obtain supplemental information • Establish evaluation methodology • Develop target survey list (USAID SO/RP staff, selected TA Contractors/Grantees, EPIs, participants, etc.) • Draft information-gathering instruments (interview guide, questionnaire, etc.) • Establish criteria for selection of Case Studies and TNAs • Begin interviews at USAID of key SO staff • Establish date for “Options Meeting” and begin sensitizing key participants 2 February 11 – 17 Information Gathering • Review Case Study files and develop interview calendar • Analyze TNAs selected • Divide interview schedule among Team Members and begin making appointments • Determine level of logistic support needed (Egyptian escort/interpreter, drivers, etc.) • Make appointments and coordinate schedules • Conduct interviews / gather data • Assess DT2 use of TraiNet • Share information gathered daily among team members at end of day • Make modifications in evaluation methodology and emphasis as needed • Propose Report Outline to USAID • Agree on writing assignments among Team members 3 February 18 – 24 Information Gathering Completed Analysis Begins • Continue with interviews and site visits to EPIs and TA Contractors/Grantees • Analyze information gathered • Synthesize, report, review and debate information gathered • Make additional appointments as needed given analysis of information obtained • Begin preparation for Options Meeting by developing an agenda in collaboration with USAID Page 88 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Week Tasks • Finalize arrangements for Options Meeting (participant list, conference room, logistics etc.) 4 February 25 - March 3 Findings & Recommendations Formu￾lated & Options Meeting Conducted • Summarize key findings in bulletized form • Summarize key recommendations • Develop presentation tables and graphs for Options Meeting and final report • Submit Options Meeting Agenda to HDD for distribution to participants • Make presentation and write summary of participant comments • Make revisions and modifications as necessary to findings and recommendations, and/or incorporate participant suggestions • Complete Team member drafts of designated sections • Leave a bulletized version of key findings and recommendations with HDD prior to departure • Depart Cairo on March 3 5 March 7 - 16 Report Finalized • Team Leader assembles all sections for review and to ensure consistency and accuracy • Team Leader completes final draft for submission to USAID Annex F Interview Questionnaire Page 90 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt DT2 Evaluation Interview Guide SO/RP No. and Title: ________________________________________________ Interviewee(s) _________________________Title/Role_____________________ Organization: ______________________________________________________ Interviewer _______________________________ Date: ___________________ Introduction (Why we are here) I am part of an evaluation team hired by USAID/ Egypt to develop findings and make recommendations about how training should be managed in the future. We therefore are assessing the various management aspects of DT2 and meeting with people who use the services of DT2. As you know, IIE is the contractor for DT2 in Egypt. We've selected a few Case Studies of training managed by DT2 and appreciate your agreeing to meet with us. We would like to ask you review the process of managing training and then its effectiveness or impact on the tar￾geted organizations. (For those who need to know, explain DT2's objectives): • (1) to foster the re-engineering of training – to ensure that it contribute as much as possible to achievement of USAID’s and Egypt’s SO/RP and • (2) to ensure that training was managed within USAID rules and regulations I would appreciate your answering some questions about DT2. Your answers will be kept confidential. I would need between 30 and 45 minutes of your time. 1. When did you start obtaining service from the DT2 project or with IIE? (NB: indicate if a person used to work with IIE and no longer does) 2A. Do you have a regular contact at IIE? Yes ______ No _______ 2B. (for USAID employees): Are there non-USAID employees on your SO or RP? Yes (indicate whether SO or RP) No If yes, how often do the teams meet? (indicatge whether SO or RP) 2C. (for non-USAID employees): Are you a member of an SO or RP team at USAID? Yes (indicate which one) No 91 If yes, how often do the teams meet? (indicatge whether SO or RP) I. Process (HAND OUT “THE TRAINING PROCESS.”) Please look at each of the five steps in the Training Process and answer the following questions. A. Determining Training Needs 3. How were the training needs determined? __ a) Large, comprehensive TNA conducted by a consultant team __ b) Interviews by IIE __ c) USAID described the needs __ d) TA Contractor determined the needs 4. Please evaluate the quality of the process by which needs were determined (1 being the worst and 5 being the best) Rating: ______________ 5. Do you have any improvements to suggest? B. Planning the Training 6. Who decided on the training objectives and types? ___ a) TA contractor ___ b) DT2/IIE ___ c) EPIs ___ d) USAID ___ e) Other (specify): 7, How was the training planned? Describe the process. Page 92 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt 8.. Please evaluate the effectiveness of the process by which training was planned. Rating: 1-5: ______ 9. Please evaluate the effectiveness of the training provider selection for in-country training: Rating: 1-5: ____ for overseas training: Rating: 1-5: ____ 10. Do you have any improvements to suggest? C. Selecting of Trainees 11. Who selected the trainees? __ a) the partner institution __ b) Technical Contractor __ c) IIE __ d) other ___________________________________________ 12, Why were the trainees selected? __ a) Key person for achieving the desired organizational changes __ b) Capacity of the trainee __ c) Political and/or relationships __ d) Rewarding people __ e) Other (specify): 13. Please evaluate the effectiveness of trainee selection. Rating: 1-5: _____ 93 14. Do you have any suggestions for improving this process? D. Processing and Monitoring Training 15. Please evaluate the effectiveness of the processing of participants going overseas for training: Rating: 1-5: ____ : 16. Please evaluate the monitoring of training while participants are overseas? Rating: 1-5: ____ : 17. Please evaluate the monitoring of in-country training? Rating: 1-5: ____ : 18. Do you have any suggestions for improving this process? E. Post-training Follow-up and Evaluation 19. Which organization took the lead in organizing follow-up and evaluation? __ a) IIE __ b) TA Contractor __ c) Training Provider __ d) Egyptian Partner Institution __ e) other ____________________________________- 20. Please evaluate the quality of follow-up: Rating 1-5: ____ evaluation: Rating 1-5: ____ Page 94 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt 21. Do you have any improvements to suggest? II. Impact 22. Evaluate the impact of the training on the partner institution: Rating: 1-5: ____ : 23. How closely linked was the training to the achievement of SO or RP results? Rating: 1-5: ___ : 24. To what extent were the training and TA integrated? Rating: 1-5: ____ 25. To what extent has the DT2 program helped to develop the capacity of local organiza￾tions (EPIs and training providers) to provide effective training? Rating 1-5: ____ : III. Overall Assessment (For all training -- overseas participant training, 3rd country and In-country) 26. How well does DT2 meet the USAID's needs for managing the overall process of train￾ing? Rating: 1-5: ____ : 95 27. How well is USAID served by a centralized source of funding for DT2, as opposed to each SO team using its own funds for training? Rating: 1-5 _____ 28. Is there more impact on the performance of partner institutions when training is managed by DT2 or by the TA contractor? 29. What lessons have you learned from DT2 that can guide the development of DT3? Page 96 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Annex G List of Persons Interviewed 97 Persons Interviewed by Institutional Affiliation USAID USAID/Cairo Mona Bawab, Office of Human Development and Democracy (HDD/PH) R. Stephen Brent; Associate Director for Human Development and Democracy Adel Gohar, COTR, University Linkages (SO 17), HDD/ET Adel Halim, SpO 18, EI/WW James Harmon, Activity Manager SpO 18, Egypt Utilities Management, EI/WW Tyler Holt, COTR, Privatization (SO 16), EG/PF Gary Kinney, Chief, Office of Procurement, PROC/OD Joan Larcom; General Development Officer and COTR, Management Devel. Initiative David McCloud, Program Officer Nagla Mostafa, Activity Manager SpO 21 NGO Service Center Fatma Naguib, Participant Training Specialist Salwa Nashed, ELTT Technical Officer Tim O’Hare, COTR, TSSPR (SO 16) Sally Patton; Chief, Human Development, HDD/ET Dr. Wadie Fahim Mankarious, Water Resources Management Specialist Dr. Nahed Matta, HDD/PH Tarek Shalter, COTR, Growth Through Globalization (SO 16) Hafiz Shaltout, EG/SP Mervat Tawfik, COTR, Capital Market Development (SO 16) Anthony Vance; Team Leader, Workforce Development James Van den Bos, Training Officer and CTO of the DT2 Project Michele Ward Brent, SpO 22 Basic Education, Girls’ Ed Advisory, USAID/CEDPA Glenn Whaley EI/EE Sami Yacoub, EI/EE Jeena Zaky, Project Officer; CID Mona Zikry; IELP-II Technical Officer, IELP-II USAID/Washington Ron Rafael; G/HCD Project Manager DT2 Training Contractor IIE/Cairo Antoun Dahdouh, Consultant, RPT Team 4 Rania Ahmed El Said, RPT Team 1 Training Specialist Dr. Yehia Gado, Training Manager Faith Galetshoge, Director of TrainingRagi Mohmoud, RPT Team 1 Leader (SO 16/17) Page 98 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Eman Mallawany, RPT Team 1 (Consultant) Samar Mohy, RPT Team 2 Leader (SO 16/22) Samia Moussa, Team Leader, RPT Team 4 (SO18/19) Melanie Sanders-Smith, Chief of Party Sherine Saber, Evaluation Specialist Azza Seif, RPT Tean 1 Training Specialist Eman Sharobeem, Team Leader, RPT Team 5 (SO 21/22) Germeen Rifky, Training Specialist, RPT Team 2 Reem Zahran, Training Specialist, RPT Team 4 IIE/Washington William Nance, Director, Global Training for Development Marjan Zanganeh, Training Placement Director, DT2 USAID Technical Assistance Contractors Abt Associates (ALEB) Doug Anderson Ghada El Henaway Alexandra Harrison Academy for Educational Development Jim McCloud, Chief of Party, Integrated English Language Program II Larraine Denakpo, Director, Program Development and Implementation Richard Panzarella, Activity Manager, Participant Training Kathy Lance, Activity Manager, English for Occupational Purposes AMIDEAST Stephen Hanchey, Activity Manager for Master Teacher Exchange Program CARANA Corp. (PFC) Derek Farwagi, Human Resources Development Specialist Center for Adult and Continuing Education, The American University of Cairo John R. Aydelott, Program Manager, (ELTTP) Chemonics International Madiha Afifi, Training Manager, Cairo Air Improvement Project Raja Sherriff, Senior Training Coordinator, Cairo Air Improvement Project Environmental Policy and Institutional Strengthening (EPIQ) Dr. Robert Cardinalli, Senior Sociologist (IRG) Dr. Zhongping Zhu, Senior Water Resources Engineer 99 Padco Farouk El Sheikh Paltech Larry Hearns, Chief of Party, MDI Project Malek Gaafar, Training Development Office John Snow International (JSI) Richard Ainsworth, Deputy Chief of Party for Management Services Dr. Reginald Gipson, Chief of Party Dr. Ali Abdel Megeid, Deputy Chief of Party for Technical Services Save the Children NGO Service Center Duncan Miller, Chief of Party Training Providers Academy for Educational Development (for IELP) Larraine Denakpo. Director, PD&I Dina Gaafar, Activity Manager Jim MccLoud, Chief of Party Helena Simas, Training of Trainers Specialist Institute for Public-Private Partnerships (IP3) Katia Karpova, Training Manger Jim Ryan, Consultant Trainer Management Sciences for Health (for APRP) Stephen Reimann, Principal Program Officer and Senior Training Officer Save the Children (SpO 21, NGO Service Center) Duncan Miller, Chief of Party PADCO (for Middle Egypt Utilities Strengthening Program, SpO 18, Egypt Utilities Man￾agement) Farouk El-Sheikh, Chief of Staff Egyptian Partner Institutions Advanced Management Institute M. N. Darwish, Assistant Dean for Academic Affairs Fayoum Economic General Authority for Water and Sanitation Mahmoud Massoud, Chairman (re SpO 18, Egypt Utilities Management) Page 100 Evaluation and Options Development for DT2 Aguirre International USAID/Egypt Ministry of Health Dr. Wagida Anwar, Senior Advisor to the Minister Dr. Esmat Mansour, Undersecretary Minya Authority for Water and Wastewater Eng. Khalil Mohammed Khalil, General Manager (Egypt Utilities Management) Water Policy Advisory Unit, Ministry of Water Resources and Irrigation Eng. Gamil Mahmoud, Manager Trainees/Participants M Nagui Darwish, Assist. Dean for Academic Affairs, Advanced Management Institute Hany El Beltagy, Assistant Managing Director, Alexandria Agriculture Heshan El Tanbouley, President, H. El Tanbouley Co. Ahmed Ghriab, Quality Manager, El Nenalea Co. Khalil Mohammed Khalil, SpO 18, Egypt Utilities Management Mahmoud Massoud, SpO 18, Egypt Utilities Management Hisham Zahran, General Manager, El Nenalea Co. Others Peter Gallagher; President, Development InfoStructure (DevIS) Norman Goodman, former IIE Director of DT2 Chris Kagy, Programmer, Development InfoStructure (DevIS) Virginia Lambert, Price Waterhouse, Consultant to USAID on MDI evaluation Diane Leach, former USAID Training Officer (now at USAID/Honduras) Jeena Zaky, Project Officer, CID, consultant to USAID on MDI evaluation 101 Annex H Documents Reviewed The list of documents reviewed is found in the Scope of Work, Pages 4-5, in Annex E.