Generations of Quiet Progress The Development Impact of U.S. Long-Term University Training on Africa from 1963 to 2003 VOLUMES I, II and III An evidence-based impact assessment of the value obtained from major investments in graduate education for 3,219 African professionals by USAID and its partners in the ATLAS and AFGRAD programs Final Report A report for: U.S. Agency for International Development Bureau for Economic Growth and Trade Education Office Washington, D.C. Prepared by: Andrew Gilboy, Team Leader Harry Carr Thierno Kane Robert Torene Consultants to Aguirre International Guided and Assisted by: Cristin Springet, USAID/EGAT/ED Prepared for the U.S. Agency for International Development by Aguirre International under the Global Evaluation and Monitoring IQC, Contract FAO-I-00-99-00010-00, Task Order 13 SEPTEMBER 2004 Generations of Quiet Progress The Development Impact of U.S. Long-Term University Training on Africa from 1963 to 2003 Volume I An evidence-based impact assessment of the value obtained from major investments in graduate education for 3,219 African professionals by USAID and its partners in the ATLAS and AFGRAD programs A report for: U.S. Agency for International Development Bureau for Economic Growth and Trade Education Office Washington, D.C. Prepared by: Andrew Gilboy, Team Leader Harry Carr Thierno Kane Robert Torene Consultants to Aguirre International Guided and Assisted by: Cristin Springet, USAID/EGAT/ED Prepared for the U.S. Agency for International Development by Aguirre International under the Global Evaluation and Monitoring IQC, Contract FAO-I-00-99-00010-00, Task Order 13 SEPTEMBER 2004 GENERATIONS OF QUIET PROGRESS The Development Impact of U.S. Long-term University Training On Africa from 1963 to 2003 USAID/EGAT/ED Table of Contents Volume I ACKNOWLEDGMENTS ........................................................................................................................ vi EXECUTIVE SUMMARY ........................................................................................................................viii Findings ...........................................................................................................................................ix Observations .................................................................................................................................xii INTRODUCTION .................................................................................................................................... xiv A. Description of the ATLAS and AFGRAD Programs..........................................................xv B. Definition of Impact...........................................................................................................xvi C. The Modified Kirkpatrick Framework...............................................................................xvi D. Organization of the Report................................................................................................xvii SECTION I: FINDINGS..........................................................................................................................1 A. Reliability of the Findings......................................................................................................1 1. Quantitative Data Collection Strategy.............................................................................1 2. Qualitative Data Collection .............................................................................................2 3. Field Visits.......................................................................................................................3 4. Internet Search.................................................................................................................3 B. Description and Analysis of Findings....................................................................................3 C. Laying the Groundwork: Analyzing Impact Using the Modified Framework.....................15 1. Acquisition of Knowledge, Skills, and Attitudes...........................................................15 2. Application of Knowledge, Skills, and Attitudes ..........................................................18 3. Measuring Changes in Institutional Output...................................................................20 D. Dissecting Impact at Higher Levels.....................................................................................22 1. At the Sectoral Level .....................................................................................................23 2. At the Community Level ...............................................................................................25 3. At the National Level ....................................................................................................27 4. At Regional and International Levels............................................................................27 E. Considering Impact by Other Variables...............................................................................29 1. The Gender Factor.........................................................................................................29 2. The Language Difference ..............................................................................................32 3. Type of Institutions........................................................................................................35 4. Significance of Degree Level and Year of Completion.................................................36 5. Individual Achievement and Career Advancement.......................................................38 F. Identifying Areas of Special Impact ....................................................................................41 1. Participant Whereabouts................................................................................................41 2. Participant Contributions outside the Workplace ..........................................................43 3. HIV/AIDS Work............................................................................................................44 Contents SECTION II: SCOPE OF THE ASSESSMENT.....................................................................................45 A. Objectives and Definitions...................................................................................................46 B. Data Collection Methodology ..............................................................................................47 1. Participant Survey..........................................................................................................48 2. Country Site Visits.........................................................................................................49 3. The Internet Search: Surfing for Impact ........................................................................50 4. Interviews......................................................................................................................51 SECTION III: LESSONS LEARNED AND CONFIRMED ABOUT U.S. LONG-TERM TRAINING ....................................................................................................................... 52 SECTION IV: PARTICIPANT FEEDBACK AND PROFESSIONAL ENHANCEMENT TRAINING ....................................................................................................................... 55 A. Ideas from the Participants...................................................................................................55 B. Professional Enhancement Activities...................................................................................56 VOLUME II: SUPPLEMENTAL BACKGROUND INFORMATION ANNEXES A. Description of ATLAS and AFGRAD Programs B. Objectives and Definition C. Modified Kirkpatrick’s Four Evaluation Levels D. Data Collection Methodology and Country Selection E. Participant Questionnaire (abbreviated version) F. Qualifications of the Data G. Internet Impact Search: Search Strategy, Findings and Recommendations H. Memo of January 11, 2004: Proposed Methodology I. Examples of Distinguished AFGRAD/ATLAS Alumni J. Scope of Work and Modifications VOLUME III: PARTICIPANT TEXT ANSWERS TO OPEN-ENDED QUESTIONS ANNEXES A. Two Most Important Examples of Acquired Knowledge, Skills, and Attitudes B. Most Important Examples of Applied Knowledge, Skills, and Attitudes C. Examples of Changes in Institutional Output D. Descriptions of Levels at Which Changes Occurred GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page iv Contents Impact Examples Impact Example 1. How Non-Technical Benefits of Training Can Yield the Greatest Impact .......................................................................................................14 Impact Example 2. Examples of KSAs Acquired during Academic Program..........................17 Impact Example 3. Examples of Skills Applied at the Workplace ...........................................19 Impact Example 4. Examples of Institutional Changes Inspired by U.S. Training ..................21 Impact Example 5. How Impact on a Sector Can Occur from Afar .........................................25 Impact Example 6. How Impact Saved Lives and Improved Health........................................28 Impact Example 7. How Independent Analysis Led to National Impact..................................40 Tables Table 1. ATLAS/AFGRAD Program Costs in Today's Dollars.............................................. xiii Table 2. Acquisition of Knowledge, Skills, and Attitudes during Training ..............................15 Table 3. Contribution of Academic Training to Professional Development .............................16 Table 4. Application of Knowledge, Skills, and Attitudes at the Work Place...........................18 Table 5. Degree of Difficulty in Applying KSAs at the Work Place.........................................19 Table 6. Location of Impact.......................................................................................................23 Table 7. Application of KSAs by Professional Field.................................................................24 Table 8. Gender Comparison of the Acquisition of KSAs during Training ..............................30 Table 9. Gender Comparison of the Degree of Difficulty in Applying KSAs...........................30 Table 10. Participant Reasons for Not Applying KSAs at Work.................................................31 Table 11. Gender Comparison of the Frequency of Accomplishments and Discoveries Cited.............................................................................................................................32 Table 12. Comparison of Extent of Application of KSAs at Work by Language Group ............32 Table 13. Comparison of the Degree of Difficulty in Applying KSAs at Work by Language Group...........................................................................................................33 Table 14. Comparison of the Location of Impact by Language Group ......................................34 Table 15. Comparison of Differences in Output by Institution Type ..........................................35 Table 16. Comparison of the Location of Impact by Degrees Obtained .....................................37 Table 17. Comparison of the Frequency of Accomplishments and Discoveries Cited by Degree Completion Year .............................................................................................38 Table 18. Examples of Current or Past Positions Held at the World Bank Group ......................43 Table 19. Participants Attending Impact Study Meetings by Country ........................................49 Table 20. Balance Achieved in Site-Visit Country Selection .....................................................50 GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page v Acknowledgments The research, data-gathering, analysis, and writing of this impact assessment took place from Oc￾tober 2003 to June 2004. A five-person team consisting of three U.S.-based and one Africa-based consultants, and the USAID Contractor Technical Representative, designed, carried out, and wrote the study. Andrew C. Gilboy, Team Leader and Principal Author of the Assessment Harry Carr, Economic Development Specialist Thierno Kane, Participant Training Specialist Cristin Springet, USAID Contractor Technical Representative Robert Torene, Data/Statistical Analyst The team members worked intermittently over the 7-month period from the United States and during four weeks of travel in Africa to seven countries. The tasks were divided into phases from design of the survey instrument and methodology through report drafting. In each of the countries visited by the team, local coordinators were engaged to locate former participants, organize a meeting/workshop, administer the questionnaire, and arrange for ap￾pointments for visiting team members with local institutions and the USAID Mission. In addi￾tion, the Ghana local coordinator (MEL Consulting) also handled all data entry for all 203 ques￾tionnaires and text entries of answers to all open-ended questions. The data could not have been compiled so cost-effectively and accurately without the invaluable help from owner/consultant Aba Quainoo and her key HR staff person, Peter Crabbe, who oversaw the development of tem￾plates and data entry. The report could not have been completed without the excellent organiza￾tional help from the following local coordinators: Benin: Flexi Management Consulting (Norbert Gohoungo and Dekadjevi Dangro) Ghana: MEL Consulting (Aba Quainoo) Madagascar: CAB PERFO Consulting, for Jimmy Ramiandrison. Mali: DJOM Consulting, Dramane Doumbia Mozambique: AAI, Celia Diniz Namibia: Doufi Namalambo Consultancy Uganda: Matthias M. Kibuka, Consultant The team embarked on an ambitious plan to contact many African alumni in the 27 countries randomly selected using all possible means in order to collect information on impact they may, or may not, have had from their long-term U.S. training. Without the additional help of the fol￾lowing people, the team would not have succeeded in gathering so much high-quality, relevant participant information about the impact of training: GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page vi Acknowledgments Roger Rasnake, Aguirre International, Inc. Tom Judy, Aguirre International, Inc. Rahama Wright, Participant Search Specialist Richard Everett, Africa Specialist, Internet Search Aba Quainoo and Peter Crabbe, MEL Consulting, Accra The team wishes to express its gratitude to all those above who worked tirelessly to find partici￾pants, organize meetings, track down questionnaires and make arrangements for this assessment. Many others took great interest and assisted as well, in particular the staff of USAID Missions, the Bureau for Economic Growth/Office of Education (G/EGAT/ED), the Africa Bureau (par￾ticularly its Office of Sustainable Development (AF/SD) and its Office of Development Planning (AF/DP)), and staff of the Africa-America Institute. And to the hundreds of African graduates of U.S. universities who voluntarily completed lengthy questionnaires and participated actively in the Africa-based alumni meetings, goes our heartfelt appreciation. As a small token our thanks, the team committed to emailing a final copy of the assessment to the 203 participants who com￾pleted the questionnaire and to any others who request the assessment. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page vii Executive Summary How many opportunities exist to assess development impact in Africa from a single program that covered 45 countries and lasted for 40 years? Established at the moment of independence for many African nations, the USAID-funded AFGRAD Program (African Graduate Fellowship Program [1963-1990]), and its successor ATLAS Program (Advanced Training for Leadership and Skills1 [1991-2003]), came to a close last April, having traversed many well-known devel￾opment challenges and obstacles. Through these four decades, the ATLAS/AFGRAD regional program, managed by the Africa-America Institute (AAI), trained over 3,200 African profession￾als for PhD and MA degrees at U.S. universities in fields critical to their country’s growth. What development impact on African institutions resulted from investing $182 million2 to bring highly qualified African professionals to the United States for graduate training? Were USAID’s development goals well served? To answer these and other questions, the USAID Africa Bu￾reau’s Sustainable Development Office commissioned a study in October, 2003 to find out whether development impact occurred from the longest-running and largest long-term graduate training programs for Africa: ATLAS and AFGRAD. This was in response to several issues: • The impending end of the program and a need to derive lessons learned from future plan￾ning; and second, questions being raised by U.S. universities, scientists and researchers, African institutions, and U.S. diplomats as to why USAID, the lead foreign-assistance agency, was not investing in replenishing the stock of U.S.-educated leaders in Africa. • Interest in sustainable and significant changes introduced by participants sponsored through these two well-known programs that received U.S. Government funding of some $182 million over 40 years; and, if so, what can be discovered about that impact? Was the impact at the individual level, or was the impact felt at the institutional, community, sec￾toral, national, regional, or international levels? • An assessment of whether USAID’s overall development objectives were well-served by such large investments. • Questions of differences in terms of participant effectiveness in bringing about positive changes linked to their U.S. academic programs by variables of gender, language, and education level. 1 Formerly the African Training for Leadership and Advanced Skills project. 2 Equal to $366 million in today's dollars when adjusted for inflation. See the last page of the Executive Summary for details. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page viii Executive Summary The results and observations discussed in the report were developed from evidence-based find￾ings, which draw from both quantitative and qualitative information. A quantitative, statistically based survey of 203 participants representing an estimated 1,921 participants, or 60 percent of the actual universe of 3,219 graduates, drives the report, supplemented by hundreds of examples from participants themselves that ground their affirmations of significant impact. The assessment team’s visits to seven of the largest “sending countries” led to site discoveries of examples where the U.S. training made the critical difference in an institution, sector or community. Even an Internet search turned up remarkable “hits” where the achievements of former participants were influencing change far beyond their country’s borders. Findings The range of findings of the assessment are highlighted here and discussed in greater detail in Section I: Findings. Finding 1: USAID’s multi-million dollar investment in long-term training for over 40 years produced significant and sustained changes that furthered African development in measurable ways. The results from all information-gathering methods used show that participants introduced many changes that made a measurable difference beyond their own lives. Over 95 percent reported making changes at their institutions and cited specific, plausible examples to verify their affirma￾tion. Change at the institutional level of this magnitude is unusual in human resources and train￾ing programs and testifies to the extraordinary impact the ATLAS/AFGRAD programs had in Africa. Finding 2: Long-term degree training at U.S. institutions was critical in creating the neces￾sary foundations for significant impact to occur. Alumni credited the “non-technical” changes they traced to their U.S. education, such as changed attitudes towards work or improved research techniques, as key to the changes they were able to introduce. These types of KSAs (Knowledge, Skills, and Attitudes) cannot be acquired in non￾degree, short-term training programs due to insufficient time to adapt to and become immersed in the U.S. learning environment, assimilate changed behavior, and allow self-confidence to flower. Finding 3: Participants reported that changes in institutional performance were attribut￾able to U.S. training and gave concrete examples as justification. The ATLAS/AFGRAD participants returned from U.S. training and applied their knowledge and skills directly in ways that had measurable impact on African institutions. There is plausible linkage between the results identified and the training obtained. Participant open-ended re￾sponses on surveys, as well as in hand-written personal impact statements, consistently linked GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page ix Executive Summary the U.S. experience with the participant’s ability to induce change (see Volume III, Annex C, Examples of Changes in Institutional Output). Since it is not possible to eliminate all influences with the exception of U.S. training on a participant’s life post academia, the assessment team utilized triangulating questions, examined written statements, and interviewed supervisors, as well as those not trained. While prevailing economic and political conditions, leadership at the institutions, and the availability of resources are often cited as factors affecting impact, without the initial long-term academic program, it is unlikely that impact would have occurred. Finding 4: Running against prevailing views, participants cited critical thinking and re￾search skills rather than improved technical and scientific knowledge more fre￾quently as critical to achieving impact. The conventional wisdom holds that were technical skills transferred effectively and sufficiently to institutions, impact would result. This view is grounded in the concept that African institutions lack technical know-how and resources that prevent their intervention in sectors to spur growth. So deep-set is this notion in both the U.S. and African organizational culture that it drives most training dollars into technical upgrading rather than into performance improvement. Finding 5: Changes in attitudes towards work consistently appeared as major benefits. Many participants developed a changed perception and strong commitment for their work during their graduate studies and credit this aspect as key to their ability to implement change (impact). Like improved research techniques and critical thinking, changed work attitudes are those “soft” by-products from technical training that are underappreciated for their contribution to impact. The research shows that factors related to organizational culture rather than scientific knowledge or professional expertise play a far more significant role in determining impact than has been previously recognized. Finding 6: No difference in impact was observed between PhD and master’s graduates. Another striking finding is that although the differences in cost are sizeable between the two de￾gree programs, no significant difference in impact was reported by PhD and master’s graduates. These quantitative results suggest that USAID’s higher investment in doctoral programs might not have yielded a higher return based on impact. Finding 7: Improved management was a frequently cited training benefit even though it received minimal attention during training. Many participants took short courses in USAID-funded mandated short management seminars during holiday or summer periods which provided them with basic management tools to facili￾tate application of their KSAs in their home institutions. When queried about the skills acquired during U.S. long-term training, participants only rarely mentioned “management.” When asked what KSAs they applied at the workplace, management skills took a prominent place. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page x Executive Summary Finding 8: Participants from the Education sector reported consistently higher impact and less difficulty applying their acquired knowledge and skills in their institutions than other sectors. Data indicated that Education sector participants found it easier than those in other sectors in ap￾plying their KSAs in the workplace. Anecdotal information gleaned from site visits, in particular interviews at universities, suggested that the emphasis placed by the program on building capac￾ity at African universities created a concentration of returned participants at key institutions. It is possible that this factor, more than any characteristic about education as a field of study, ac￾counted for a more supportive organizational environment for application of KSAs. Finding 9: Participants with degrees in Financial fields, or those with MBAs, recorded lower impact than those in Agriculture, Health and Education. Although their numbers were low, participants in the financial sector and those with MBA de￾grees registered lower levels of impact than the three academic fields that predominate in both ATLAS and AFGRAD—agriculture, education, and health. While the significance is diminished overall by the relatively small proportion of graduates in these two fields, compared to the larger fields, it is a surprising finding that bears further inquiry beyond scope of this assessment. Finding 10: Although women reported more difficulty applying their knowledge and skills at the workplace than men, they reported impressive anecdotal examples of impact where they were able to apply their skills and knowledge. In surveys, female participants reported fewer achievements than their male counterparts— primarily because of more difficulty applying their newly acquired knowledge and skills at the workplace. Anecdotal data, however, show that women found ways to overcome these impedi￾ments to excel in ways unimaginable before training, citing mainly “increased self-confidence.” Finding 11: No correlation could be found regarding impact and the frequency with which participants returned to their original workplace. While participants reporting impact often worked in different institutions or even worked out of their country for periods, nearly half of those surveyed (49.4 percent) did return to work at the same institution where they were employed prior to their program in the United States. This ap￾pears to be an impressively high number, given the ups and downs of many African institutions over the years and the political instability that thwarted the return of many participants (to Ethio￾pia, Uganda, Ghana, Liberia) for years. The percentage includes recent returnees as well as re￾tired alumni. Finding 12: Participants returned to their home countries after their U.S. training when conditions permitted. There is no significant evidence that long-term U.S. train￾ing under these sponsored programs contributed to any brain drain of African human resources. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xi Executive Summary It was not in the scope of this review to verify AAI’s claim that an average of between 85 and 90 percent of participants from all programs over the years returned to their country—an impres￾sively high rate of participant return for any long-term U.S. training program. The issue of rates of return is more complex than is generally thought and requires careful definition of terms used (such as “return”). The fact that the two programs stretched over such a long period makes the rate of return far more credible as a measure of return (not a measure of impact) since it flattens out the fluctuating security and economic conditions of African countries over decades. It is also true that a participant who resides overseas at the time of this assessment in no way implies lack of impact or even that the person did not return to the home country sometime after training. The assessment team encountered little criticism of the return rate from targeted institutions Finding 13: ATLAS/AFGRAD participants surveyed were well-advanced in their careers, making significant contributions to development. Participants were selected for ATLAS/AFGRAD generally after some work experience, in col￾laboration with their employing institution, and were often mature professionals prior to starting a graduate education in the United States. These considerations often promote closer linkages between the academic program the participant undertakes and the development needs of the sending institution or country. Examples of these significant contributions are cited in Volume III, Annex C. Observations In the course of examining the impact from the ATLAS/AFGRAD programs, the assessment team developed findings that deflate many of the myths that typically taint long-term training initiatives for overseas students as a valuable development tool. For instance, widespread percep￾tions hold that U.S. government–funded long-term training goes to children of the well-off Afri￾can elite, participants do not return home, scarce training funds would be better spent on more trainees in-country, and brain-drain is worsened. Although the study was not designed to address these myths, it indirectly counters them by uncovering significant contributions made by partici￾pants across the spectrum of fields affecting economic growth in Africa. • First, the program brought employees of key institutions who had work experience to the U.S., not elite, hand-picked students or “best and brightest” undergraduates. • Second, roughly 90 percent of the participants returned home, when conditions al￾lowed—it was uncommon for a participant to be unemployed. • Third, the cost per impact derived from USAID’s investment may well be lower for high￾performing, impact-producing participants trained at U.S. universities than for those trained in-country when compared accurately. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xii Executive Summary • Fourth, brain drain was contained—not worsened—by the major contributions partici￾pants made in their home-country institutions and sectors that multiplied opportunities, improved the learning environment, and raised hopes for young, upcoming professionals. A significant volume of quantitative and qualitative data was collected during the course of the assessment, only a part of which could be exploited for this study. This valuable information is available in electronic and hand-written form and merits further analysis to increase understand￾ing about impact derived from long-term graduate education. That African development has been halting and somewhat disappointing should not diminish the significant accomplishments of the participants and the program. A more pertinent question might be what impact could have occurred had the relatively small cost of the long-term pro￾grams (some $4.5 million a year) been multiplied tenfold, or a hundred-fold? Table 1. ATLAS/AFGRAD Program Costs in Today's Dollars Program Funding ($) CPI Multiplier 2003 ($) ATLAS (1995) 75,383,795 152.4 1.207 90,888,240 AFGRAD III (1985) 53,600,615 107.6 1.710 91,657,051 AFGRAD I & II (1975) 53,600,615 53.8 3.420 183,314,100 TOTAL 182,585,026 365,959,391 Note: This table shows the calculation used to arrive at the amount that would be required in 2004 to fund the same program, adjusted for inflation, and assuming tuition waivers from U.S. universities. The Consumer Price Index was determined to be the fairest method to arrive at the adjusted amount since most of the program costs were monthly living allowances, travel and ad￾ministrative overhead—all well represented in the CPI. A single year was selected mid-way between each program's implementa￾tion to use as a base year. Another calculation used was based solely on tuition increases over the years that resulted in a $713 million price tag in today's dollars. The two calculations can be used to estimate what a new long-term training program would cost today, with and without tuition. On a per-year basis, the $4.5 million per month program cost cited ($182 million / 40 years) would be doubled to $9 million to replicate a new program with tuition waivers, and considerably more with no waivers. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xiii Introduction The ATLAS and AFGRAD3 programs were together among the largest and longest-running, long-term training initiatives of their kind in Sub-Saharan Africa. For years the notion that higher education was “a good investment” to help advance developing country growth was sufficient justification to secure large amounts of U.S. government funding for long-term training for insti￾tutions in Latin America, Africa, and key countries in Asia (in particular, South Korea). This ra￾tionale and sizeable funding have withered to the extent that today, few USAID programs sup￾port institutional capacity-building through graduate training in key development fields, espe￾cially for African professionals. This report seeks to answer the interests and questions raised in the Executive Summary about developmental impact from graduate training programs that reached over 3,000 African profes￾sionals. The report draws on the experiences of impact assessments conducted in the 1990s by USAID that sought to quantify development impact in a new way.4 It focuses on the impact de￾rived from these investments in building capacity rather than the inputs used to achieve results. The assessment did not seek to explain the reasons that funding for long-term training for Africa declined substantially for the most part in the 1990s. Nor does it set out to evaluate the manage￾ment of the program by the contractor, an exercise that could lead to implementation improve￾ments but would shed no light for policymakers on the value of long-term training. Also ex￾cluded from the scope of this study is any rigorous comparison of the two programs under review (ATLAS and AFGRAD) or of different types of programs (centrally funded, regional, bilateral, sector-based, long-term versus short-term, U.S. versus third-country, and so forth) used by USAID over the years. The report is addressed to a wide-ranging audience of policymakers, program managers, and educators who need information based on solid evidence on effective programs in order to design future programs (and budgets) to address developmental and U.S. interests. This list includes: • policymakers working in development agencies; • congressional oversight committees staff; 3 ATLAS—Advanced Training for Leadership and Skills (formerly African Training for Leadership and Advanced Skills) project, operating in two phases with approximately 10 years of “intakes” from 1991 to 2003; AFGRAD— African Graduate Fellowship Program, with three phases of “intakes” from 1963 to 1990. Both projects were man￾aged by the Africa-America Institute (formerly, the African-American Institute). 4 The Human Resources Development Assistance (HRDA) project led the way in developing a methodology to quantify impact assessments for Africa throughout the 1990s, resulting in the publication of a 9-part guide Best Practices Guide for Results-Oriented Training. Similar advances were made in assessing the impact from the CLASP and CAPS projects in Latin America. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xiv Introduction • Africa specialists; • human capacity development experts; • U.S. scientists and professionals seeking African colleagues with whom to collaborate on site-specific research in agriculture, HIV/AIDS, health and education; and • U.S. diplomats faced with a declining number of U.S.-educated interlocutors at overseas posts. Because the findings were so well-founded and illuminating, this report furnishes concrete evi￾dence on the impact obtained from U.S. long-term training that can enlighten these and other groups involved in the complex business of promoting growth in developing countries. A. Description of the ATLAS and AFGRAD Programs The goal of AFGRAD program in the early years was oriented toward assisting young African nations with a supply of trained mid- and upper-level “manpower” in key sectors needed for de￾velopment. In many cases, U.S. graduates replaced expatriates in key public-sector institutions. Subsequent iterations of the goal emphasized assisting African institutions build capacity. ATLAS continued this trend, introducing concepts of improving institutional performance and broadening the target institutions to include nongovernmental organizations (private companies, NGOs, and so forth). ATLAS also added the notion of leadership development to its purpose, and its name. The combined “waves” of USAID-sponsored graduate students administered through the AT￾LAS and AFGRAD programs ebbed and flowed for over 40 years. Through the three phases of AFGRAD and two of ATLAS, the Africa-America Institute managed the selection, placement, orientation, monitoring and follow-on for 3,263 Africans from 52 countries.5 Over the years, these regional programs were occasionally evaluated employing a methodology that focused on inputs (such as the number of participants receiving advanced degrees) or the whereabouts of graduates. The last evaluation completed in 1995, which was designed as a fol￾low-up to a 1983 study, took a step further and reviewed participant impact based on a partici￾pant survey and small focus groups organized in four countries. This is the first assessment of the programs since the 1995 study. The total USAID funding for both AFGRAD and ATLAS was $182,585,026.6 Funding for AFGRAD I, II, and III was $107,201,231 and for ATLAS I and II $75,383.795. The U.S. Gov￾ernment portion of the total cost of both programs is estimated to be 85 percent, with the remain- 5 The total participant entries in the AAI database for ATLAS and AFGRAD were 3,263 from 52 countries. The assessment team excluded from consideration 20 participants who did not complete graduate degrees, and 24 par￾ticipants from outside Sub-Saharan Africa (Tunisia, Morocco, and several Caribbean nations), for whom awards were made for various reasons, to arrive at 3,219 participants for the survey. 6 The AFGRAD I and II accounted for roughly half of the three phases. ATLAS I was $27,455.214 and ATLAS II was $47,298,581. (The actual amount for ATLAS II will be slightly higher when closed out in 2004.) GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xv Introduction ing 15 percent paid by U.S. colleges and universities that waived tuition and African institutions that continued salary payments to participants while in training. B. Definition of Impact Since the term impact is open to various interpretations, the assessment team, prior to the as￾sessment design, decided on a definition of impact to apply throughout the study. Impact is any change that occurred at the institutional, sectoral, community, national, or regional level attributed to ATLAS/AFGRAD-sponsored training. The Scope of Work also requested that the study “draw conclusions about the contribution of long-term U.S. participant training in achieving USAID’s broader development objective.” This challenge fit neatly into a framework that would determine whether impact occurred that was directly related to the long-term training the participants received. By focusing on impact, as de￾fined above, and as identified through information-gathering, the assessment team could provide insights to reaching “USAID’s broader development objective.” The intent is to limit the scope so that a more in-depth look at whether USAID’s investment in long-term training resulted in any impact. (See Volume II, Annexes B and D for greater detail on the objectives of the assess￾ment and methodologies employed.) C. The Modified Kirkpatrick Framework In view of the program’s emphasis on academic training, the assessment team decided to employ Kirkpatrick’s evaluation levels as a useful framework to guide the assessment. Donald Kirkpatrick’s Four Levels of Evaluation, created in 1959, allows assessment teams to view im￾pact from training through a prism that gives structure and soundness to the journey. The time￾tested framework is simple yet adaptable to a variety of settings, including developing countries. The evaluation “hierarchy”—with the individual at its base and the institution at its peak— supplies a tool to begin the search for impact from training and allows for add-ons and adapta￾tions along the way. The Kirkpatrick methodology easily guides the development of a concise and targeted survey instrument by enabling the designers to aim questions at four levels. Change attributed to training at Levels 2, 3 or 4 is impact. However, this assessment concentrates on Levels 3 and 4 because impact at Level 2 is limited to individual rather than institutional change. • Level 1 Reaction The trainee’s impression of the program; the level of satisfaction with the course, trainer, pace of instruction, content and materials; • Level 2 Learning The acquisition of skills and knowledge from the training; • Level 3 Application The performance of the trainee on the job following training; and GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xvi Introduction • Level 4 Results Changes that the trainee’s performance brought to the organization in efficiency, productivity or profitability. In order to adapt the Kirkpatrick framework to the African setting, and in particular to assess im￾pact from long-term training over 40 years, the assessment team decided to modify the model to expand the areas where impact might occur, adding a top level to include sectoral, community, regional, national, and international in order to capture impact outside an institution—far beyond a single organization. The Modified Kirkpatrick Evaluation Model is presented in greater detail in Volume II, Annex C of this report. D. Organization of the Report The report is divided into three volumes: Findings, Background Information, and Text Answers to Open-ended Questions. • Volume I: Findings, Scope of the Assessment, and Policy and Planning Observations consists of three sections. Section I provides a summary and brief explanation of each finding followed by analysis of both quantitative and qualitative data organized according to Kirkpatrick’s four levels of impact, the assessment team’s modified higher level im￾pact, and “other” variables on impact (gender, language, type of institution, education level). Section II describes the methodology used to gather quantitative and anecdotal in￾formation and on the reliability of the findings. Section III contains planning and policy observations to be considered in the development of any future long-term training pro￾grams—ideas which emanated from this study and have been noted over time in similar long-term training programs. • Volume II: Supplementary Background Information includes detailed descriptions about each aspect of the methodology employed, including steps followed and tables showing the various survey universes, and in selecting site-visit countries. The volume contains the Kirkpatrick Model, a participant questionnaire, an Internet Search Strategy that was employed, modifications to the Scope of Work, and some examples of distin￾guished ATLAS/AFGRAD alumni. • Volume III: Participant Text Answers to Open-Ended Questions is available as a separate document. It contains examples of acquired KSAs, applied KSAs, changes in in￾stitutional output, descriptions of levels at which changes occurred, and statements or achievements, discoveries, or other contributions. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page xvii Section I: Findings A. Reliability of the Findings Findings about impact from human resource investments do not lend themselves to the methodo￾logical threshold expected in scientific laboratory experiments. The key objective in assessing human impact is to examine plausible links between intervention (training) and impact (results). Attributing the intervention to a specific observed outcome is always problematic with long-term training given the number of variables at play. But convincing assertions based on a variety of evidence can lead to evidence-driven findings that training made the difference. It is this objec￾tive that this assessment strives to reach. This assessment made important strides in collecting information in a systematic manner and us￾ing statistical analysis where possible to enhance the credibility of its findings. It employed a modified impact framework to guide both data collection and analysis. Taking into account both the limitations as well as the strengths of the sampling and the rich discoveries from the anecdo￾tal information-gathering, the findings reflect a clear, persuasive picture of the impact generated by the participants sponsored for long-term academic training at U.S. universities by the USAID￾funded ATLAS/AFGRAD programs. This study collected a combination of quantitative and qualitative data from participants, their supervisors, and colleagues that could be analyzed to determine whether impact occurred that could be attributed to the long-term academic training they received in the United States. By ac￾cessing data of different statistical reliability and type (for example, quantitative survey data with in-country participant impact statements), the credibility of the findings increased (see Volume II, Annex D for a more detailed description). 1. Quantitative Data Collection Strategy In regards to the participant survey and the methodology described elsewhere in this report, the following observations can be made: • Quantitative data was culled from a weighted sampling from 21 countries representing 1,921 (60 percent) of the 3,219 actual participants of the program. • Since the data represents about 60 percent of the total universe, the remainder of the uni￾verse can be presumed to have similar characteristics. • Quantitative data was drawn from a sampling that was stratified to ensure proportional representation by sex, region, and program (ATLAS/AFGRAD). GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 1 Section I: Findings • The sample was drawn in a way that allows for a calculation of the reliability of the data (relative standard error) shown in each of the data tables that follow. By noting the standard of error in each data table presented in this report, the reader can assess the reliability of the analysis drawn from the quantitative data presented.7 The findings based on data that have low relative margins of error for the 21 countries surveyed. 2. Qualitative Data Collection The participant survey also gathered text answers to open-ended questions. • Qualitative data was then taken from the survey instrument in a systematic fashion that encouraged quasi-quantitative analysis of participant-generated text. By grouping the text answers by similarities, and ranking them by the frequency with which they were cited freely by participants, the team was able to analyze trends and evaluate examples. • Although the report scope called for organizing small participant “focus groups,” the team decided to reformulate the mechanism and, instead, conduct more structured meet￾ings with a larger number of “key informants” in the style of a participatory workshop. The workshop approach allowed the team to challenge a group of perhaps 15 to 25 par￾ticipants to explore what impact they might have had together, in small work groups, to produce more tangible findings than the outcome from a focus group. After the team pre￾senters led an interactive discussion of the assessment’s objective methodology discus￾sions, small work groups were formed and instructed to (a) write down a description of impact, if any, each participant believed to have had that was attributed to the training re￾ceived; (b) present one-by-one these statements to the small group; and (c) select one statement to present to the other participants in the plenary session. The result was a col￾lection of invaluable hand-written statements from each U.S.-educated participant that described precisely, often many years after returning home, the changes that occurred linked to the ATLAS/AFGRAD program. • In the seven countries visited, team members met with decision-makers at local institu￾tions who were able to comment on whether changes occurred after a participant’s return that could be attributable to long-term training. In some instances, the team was able to interview a supervisor who could give impressions of performance before and after re￾ceiving the training, a rare opportunity in the fast-changing organizational context of Af￾rican institutions. Team members were able in some cases to visit laboratories or research stations that were begun by former participants. • Based on the first two site visits to Mozambique and Namibia, the team decided to select in each country one or two outstanding alumni with extraordinary stories about changes 7 Standard error is described in the Annexes (Qualifications of the Data) with several examples to assist those unfa￾miliar with statistical analysis in understanding how it is used. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 2 Section I: Findings they attributed to their U.S. training. The objective was to probe more deeply into the characteristics of this impact and the obstacles the participant overcame. After each meet￾ing, the team set up follow-up appointments with the individuals, and team members pro￾duced short descriptions of these “special impact stories” for the record. 3. Field Visits The process employed by the team to select the seven countries visited (Benin, Ghana, Madagas￾car, Mali, Mozambique, Namibia, and Uganda) ensured a level of reasonable balance in collect￾ing anecdotal information from the field. To reduce bias, the local coordinators (with one excep￾tion8 ) hired to handle in-country arrangements and co-facilitate participant meetings had no in￾volvement with the management of the ATLAS/AFGRAD programs. The team interacted with 136 participants in workshop settings, and interviewed dozens of others in individual sessions during the site visits. Participants at these meetings wrote personal “impact statements” which provided even more glimpses of the level and quality of impact in the words of the U.S. gradu￾ates themselves. In sum, the country visits greatly enhanced the quality of the findings. 4. Internet Search A randomized, statistically based, Internet search approach was used that enriched the quality of findings as well. Out of a random sample of 100 ATLAS/AFGRAD participants, the Internet search resulted in 69 matches. Fifty-one of these hits, fully half of the sample, were clearly iden￾tified as participants. To determine whether the hit qualified for an indication of impact, the re￾searcher looked for any one of the following: research papers, commissioned reports, or books written by participants; participation in national or international conferences, as evidenced by conference programs, proceedings, or lists of participants or speakers; listings in university or international organization directors; or news stories quoting or profiling participants’ activities. The researcher then compared information associated with the name of the hit with the ATLAS/AFGRAD database to determine if the hit was in fact the participant in question. When there was a match, the researcher summarized available information on current or previous job titles, papers, or books produced and made note in an Excel spreadsheet. B. Description and Analysis of Findings This section presents the assessment findings supported by information gathered via the data col￾lection instruments described in Volume II and the Annexes. Finding 1: USAID’s multi-million dollar investment in long-term training for over 40 years produced significant and sustained changes that furthered African devel￾opment in measurable ways. 8 In Mozambique, the AAI representative handled local arrangements. In all the other countries, the team worked through independent local firms and consultants. AAI representatives did not participate in in-country meetings. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 3 Section I: Findings The results from all information-gathering methods used in the study show that participants in￾troduced many changes that made a measurable difference beyond their own lives. Over 95 per￾cent reported to have made changes at their institutions and cited specific, plausible examples to verify their affirmation. Because the methodology employed verifies and triangulates answers and reduces inherent bias in participant “self-perceptions” of impact, this finding is significant and supported by evidence. Change at the institutional level of this magnitude is unusual in hu￾man resources and training programs and testifies to the extraordinary impact the ATLAS/AFGRAD programs had in Africa. In order to induce change at the institutional level, participants had first to acquire knowledge, skills, and new attitudes (KSAs), and then apply them effectively at the workplace. At these lev￾els, there is convincing evidence that changes took place. What is needed, however, for USAID to observe a return on its investment, is a measurable change in an institution’s output or produc￾tivity. Without a higher change, work-place improvements could remain intra-organizational. The assessment tracked countless changes that were made at the institutional level by partici￾pants from all countries and in all sectors. The range was impressive, from developing a new lo￾cally adapted cowpea variety that improved farm yields, to creating a chemical testing laboratory to control the importation of dangerous drugs and foodstuffs that saved unknown numbers of lives. Annex C in Volume III of this report contains many examples written in the participants’ words that convey the extent of the impact of these two programs. If such noteworthy advances were made at institutions, was impact recorded at other levels, such as in the community, nationally, regionally, or internationally? Here again, significant examples abound from survey responses, participant meetings, and Internet searches. After their institu￾tions, participants selected “nationally” as the next level at which their impact occurred—and provided many examples that grounded their claim. The Internet search also confirmed a surpris￾ing number of internationally recognized participants—some 51 percent of the 100 names drawn randomly from the master participant list passed the threshold of “impact” established for a web hit. Their contributions were judged to be felt internationally, regionally, and nationally. It is neither necessary nor realistic to attempt to derive a yield on USAID’s investment in eco￾nomic terms of 40 years of U.S. training for African professionals. It is abundantly clear that thousands of examples of impact over the period produced immeasurable changes that advanced development across the African continent. Impact happened from participants acting at a variety of levels of responsibility as well, depreciating the notion that high positions either imply impact or are prerequisites for it. How much change resulted from a single university biochemistry pro￾fessor whose enlightened teaching methods, and improved technical knowledge, influenced posi￾tively thousands of university students over a 30-year span? Or the woman lawyer who created an NGO that increased awareness of, and advocacy for, civic and human rights among thousands of village-based women in only a few years? When impact with such potency is documented, the size of the individual investment in a long￾term U.S. degree is brought into focus. Instead of comparing the $50,000 to $100,000 needed for GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 4 Section I: Findings a single academic program in the United States with the cost of training 30 computer program￾mers in a program in their country to computerize financial management, program planners should estimate the value of the anticipated impact to be generated by the trained participant. Us￾ing the example above, were program planners to consider the cost of obtaining the impact rather than the cost of obtaining the training, the return on having thousands of women in villages trained year after year about HIV/AIDS or their marriage rights because one participant acquired a KSA, applied it, and made a difference, would far exceed the return from the junior accountant program. The ATLAS/AFGRAD programs created dynamic catalysts for development, most of whom in turn made a remarkable difference in the lives of thousands of people, in institutions, and far be￾yond. If spread evenly over the years of the ATLAS/AFGRAD programs, the cost to the U. S. government in actual dollars averaged approximately $4.5 million per year. Finding 2: Long-term degree training at U.S. institutions was critical in creating the neces￾sary foundations for significant impact to occur. Participants highlighted the “non-technical” changes they traced to their U.S. education, such as changed attitudes towards work or improved research techniques, as key to the changes they were able to introduce. These types of KSAs cannot be acquired in non-degree, short-term train￾ing programs. They require immersion in the U.S. learning environment so that participants can assimilate changed behavior and allow self-confidence to flower. The characteristics of change noted during the course of information-gathering confirm that short-term exposure to the United States may be insufficient. A well-structured one-month U.S. visit with a technical objective can result in a significant impact on a participant’s institution or career, such as the transfer of an idea or the creation of a new network that ensures future access to improvements. But the attitudinal changes regarding work or performance, and advances in critical thinking, cannot be expected to be rooted over a busy one-month U.S. study tour. In terms of in-country training, the lack of exposure to the outside, and preoccupation with family and work needs, might matter much more than technical transfer of know-how. Finding 3: Participants reported that changes in institutional performance were attribut￾able to U.S. training and gave concrete examples as justification. The ATLAS/AFGRAD programs aspired to increase the capacity of African institutions to effect development changes. Participants returned and applied their knowledge and skills directly in ways that had measurable impact on African institutions. But was the impact due to the received training? This persistent challenge to all impact assessments and training assessments has no easy answer. Although it is accepted that human resource specialists do not need to show cause and effect with regards to training and impact, there should be some plausible linkage between the result identi￾fied and the training obtained. Participant responses in the text answers on the questionnaire, as GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 5 Section I: Findings well as in the hand-written personal impact statements produced during the site visits, consis￾tently linked the U.S. experience with the participant’s ability to induce change. Since it is not possible to eliminate all influences with the exception of U.S. training on a participant’s life post academia, the evaluator is reduced to triangulating questions, examining written statements, and interviewing supervisors and those not trained. A reading of the examples given (see Volume III, Annexes A and B) leaves no doubt that U.S. training played the most significant role in producing the changes cited by participants. Would the impact have occurred without U.S. training? Most participants would agree that the experi￾ence in the United States was critical. Were there other variables that assisted in obtaining im￾pact? Absolutely—according to most respondents, who frequently indicated other factors at work. Prevailing economic and political conditions, leadership at the institutions, and the avail￾ability of resources are often cited as factors affecting impact. But the underlying assumption is that without the initial academic program, little impact would have occurred. Finding 4: Running against prevailing views, participants cited critical thinking and re￾search skills rather than improved technical and scientific knowledge more fre￾quently as critical to achieving impact. The conventional wisdom holds that if technical skills were transferred effectively and suffi￾ciently to institutions, impact would result. This view is grounded in the concept that African in￾stitutions lack technical know-how and resources that prevent their intervention in sectors to spur growth. So deep-set is this notion in both the U.S. and African organizational culture that it drives most training dollars into technical upgrading rather than into performance improvement. Close to two-thirds of all amounts spent for training in the United States are reportedly spent to upgrade staff competencies, yet two-thirds of the constraints to improved performance cited by employees are managerial not technical. This alarming disconnect extends to Africa as well, where acquiring technical skills is viewed as critical to increasing productivity and breaking down organizational hierarchies through team building is considered an irrelevant luxury. Many will be surprised that, time and time again, participants pointed to changes in their ability to think critically, design research projects correctly, and analyze objectively as more than or as important as the specific technical knowledge they gained in the United States. Although a num￾ber of assessments have highlighted similar observations drawn from participant interviews and focus groups, this assessment discovered and quantified this finding. The table quantifying the participant responses (see Examples of KSAs Acquired during Academic Program, page 17) pro￾vides the details. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 6 Section I: Findings Finding 5: Changes in attitudes towards work consistently appeared as major benefits. KSAs Acquired During Training: Comments from Participants • I learned strategic thinking and planning. • I sharpened my skills in road design. • I became a life-long learner. • I improved my communication skills. • My confidence grew a lot. • I learned to respect others’ views and ideas. Although few participants studied “work atti￾tudes,” many linked their changed perception of the importance of their work to their U.S. ex￾perience. It is clear that many participants de￾veloped a strong commitment for their work during their graduate studies and credit this as￾pect as key to their ability to implement change (impact). Only in the lists of participant exam￾ples (Volume III, Annex A) of acquired skills does “work attitudes” appear prominently. That the participants themselves repeatedly stated this and similar citations as examples of KSAs ac￾quired and applied lends far more credibility to the finding than if respondents had checked off “work attitudes” on a questionnaire from a list of potential attitudes assimilated. Here is a case where the qualitative, open-ended response yields more value to the assessment than the quanti￾tative. An analysis (see Impact Example 2: Examples of KSAs Acquired during Academic Pro￾gram, page 17) shows that “work attitudes” was selected as third in a priority ranking by partici￾pants—above Computers and Management. Like improved research techniques and critical thinking, changed work attitudes are those “soft” by-products from technical training that are underappreciated for their contribution to impact. Clearly, factors related to organizational culture rather than scientific knowledge or professional expertise played a far more significant role in determining impact than has been previously rec￾ognized. The Nova Knits (Madagascar) text box on page 14 illustrates the value of attitudinal changes over strict technical competency. The fact that a PhD in physics is not teaching at the national university (although he did help establish the department upon his return) in no way has dimin￾ished the value of the development contributions made by the participant. Although no program manager could have anticipated exactly how the impact would occur with Charles or Albain, the participants might have been interviewed with an eye to observing character traits as well as in￾tellectual ability. In the end, the former turned out to be key to their assimilation of non-technical aspects of applying knowledge to solve human problems. Finding 6: No difference in impact was observed between PhD and master’s graduates. Another striking finding is that although the differences in cost are sizeable between the two de￾gree programs, no significant difference in impact was reported by PhD and MA graduates. These quantitative results suggest that USAID’s higher investment in doctoral programs might not have yielded a higher return based on impact. If a $70,000 investment, for example, in a mas￾ter’s degree led to the same reported impact as the $150,000 investment for a doctoral degree, USAID institutional capacity-building specialists may want to factor in the “cost-by-impact” when planning for institutional strengthening. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 7 Section I: Findings When the data were analyzed by sector (“field of study”), a small difference appeared indicating that Agriculture was singled out for “no impact” by a relatively high number of PhD holders (al￾though the number was small in absolute terms). In contrast, all MA graduates in Agriculture claimed to have had impact at the institutional level. Given the small size of the sampling, no finding can be deduced about Agriculture based on the quantitative data alone. In fact, the anecdotal information showed that Agriculture was one sector that boasted countless “success stories” and verifiable examples of significant change. But the principal finding above is less about the sector with the most impact, and more about the degree that is associated with the most impact. Is it possible that the value of the changes introduced by MA participants could equal or even exceed that of the PhD holders? If that were possible, plan￾ners would be hard-pressed to justify USAID sponsorship of all but the exceptional case for a PhD. Enough questions are raised by this finding for program managers and decision-makers to be ad￾vised to carefully address the issue of MA versus PhD degrees in future long-term training pro￾grams. The cost differentials are so great that the added value of a PhD must be weighed and jus￾tified with great care. “Degree-creep” in long-term training (when MA degree participants and their faculty advisors pressure sponsors to continue funding through the doctorate because the participant has excelled) can sap up much-needed funds for other MA candidates waiting in the sidelines whose contributions to development may be every bit as valuable as their highly es￾teemed superiors. No one discounts the value added of PhD professors and researchers at na￾tional or regional institutions. The questions are how to plan and allocate scarce funds according to anticipated development impact, and where the investment will have the greatest impact value. Finding 7: Improved management was a frequently cited training benefit even though it received minimal attention during training. Many participants took mandated short management seminars during holiday or summer periods that USAID funded to provide participants with basic management tools to facilitate application of their KSAs in their home institutions. But when queried about the skills acquired during U.S. long-term training, participants infrequently cited “management,” opting instead for the knowl￾edge gained in conducting research (discussed above) or in their technical area, which would be expected given the academic nature of the graduate studies. However, when asked what KSAs they applied at the workplace, management skills took a prominent place. A participant from Madagascar, who taught at the university level in the United States before returning, provided specific examples of management improvements that were quite apart from the technical exper￾tise gained. I introduced a “results-oriented” way of thinking within my organization. As a result, work is getting done much faster. My U.S. experience influenced me to have a business-oriented way of thinking in the computer businesses I es￾tablished. Also, I applied the “American” way of teaching to my computer GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 8 Section I: Findings science students here in Madagascar. The result is that they love it and de￾mand more. In the case of a participant from Namibia, who received a graduate degree in management, the application of new systems and work cultures made a measurable difference. The one important lesson/knowledge which I acquired from my program was the strategic alignment of your Human Resources strategy with your business strategy. The HR (Human Resource) specialist should have a very good un￾derstanding of the business (not just his/her specialty) and should be able to talk figures—that is, show that what he or she does is measurable. The changes produced have led to tangible evidence such as changes in university policy, structure and statutes. Management tools were transferred extensively by ATLAS/AFGRAD participants to their work environments as byproducts of the U.S. experience and as technical expertise acquired through an academic program. The evidence for this finding comes more from the number and quality of text examples put forward by participants in their questionnaires, individual impact statements, from group meetings and through interviews. The quantitative data was of secondary value in that it did not delve into the characteristics of the impact of management. Finding 8: Participants from the Education sector reported consistently higher impact and less difficulty applying their acquired knowledge and skills in their institutions than other sectors. Data indicate that Education sector participants found it easier than those in agriculture in apply￾ing their KSAs in the workplace. In terms of the supportive organizational environment, Health graduates encountered considerably more difficulty than either Agriculture or Education, with 51 percent of the survey respondents from that sector indicating changes were “possible but diffi￾cult” (compared to 57 percent of the Education graduates selecting “very easy”). By far those in the financial sector fared the worst, with 81 percent reporting difficulties. Anecdotal information gleaned from site visits, in particular interviews at universities, suggested that the emphasis place by the programs, especially AFGRAD, on building capacity at African universities created a concentration of returned participants at key institutions. It is possible that this factor, more than any characteristic about the Education field of study, accounted for a more supportive organizational environment for application of KSAs. That agriculture institutions ap￾peared to be less conducive to change induced by participants was not substantiated by extensive anecdotal information suggesting that impact was high in the sector. The financial sector did ap￾pear to present particular difficulties, but the number of participants in that sample subset, or even as a proportion of the 3,219 participants, was small. The concept of “critical mass” as a precondition for organizational change came and went throughout the implementation of the two programs. It was widely touted in the 1990s as the way GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 9 Section I: Findings to move public-sector institutions from lethargy to innovation, and many in-country training pro￾grams targeted large numbers of mid-level civil servants for repeated training programs. Higher￾level decision-makers traveled to the United States for training during the period when programs such as the ones at Pittsburgh and Atlanta equipped thousands of Africans with modern man￾agement tools to supplement their sector expertise, which created critical masses of “like￾minded” professionals. Local support associations were even established. The long-term gains from the changes introduced as a result of “critical mass” short-term training was eventually compromised by the exodus of the trained cadres from the public service in the 1990s. In con￾trast, the concept worked well in the stable, low-turnover human resources environments found at universities and research institutes and provided long-term graduates with the supportive envi￾ronment in which they could introduce and sustain changes. Finding 9: Participants with degrees in Financial fields, or those with MBAs, recorded lower impact than those in Agriculture, Health, and Education. Although their numbers were low, participants in the financial sector and those with MBA degrees registered lower levels of impact than the three academic fields that predominate in both ATLAS and AFGRAD—agriculture, education, and health. Although the quantitative data sup￾port this finding, the significance is diminished overall by the relatively small proportion of graduates in these two fields, compared to the other three. It is nonetheless a surprising finding that bears further inquiry but is regrettably beyond the scope of this assessment. Given that the MBA as a development-related field emerged primarily under ATLAS after the 1990s, with great expectations of impact in tandem with the push for private-sector programming, what would ex￾plain the suggestion that the benefits from USAID’s investments were less than in the more typi￾cal development-oriented fields? Did financial sector and MBA participants find less-supportive work environments when they returned? Were the degree programs not responding directly to institutional needs, in contrast to the clear need for increased human capacity at agricultural, educational, and health institutions? The assessment is unable to make a determination as to the reasons for this finding due to the lack of specific information that could shed further light. Finding 10: Although women reported more difficulty applying their knowledge and skills at the workplace than men, they reported impressive anecdotal examples of impact where they were able to apply their skills and knowledge. In terms of acquisition of knowledge and skills, the prerequisite for their application at the work￾place, the data showed no difference between women and men. At the application phase, how￾ever, survey results indicated that women encounter resistance to introducing changes at the workplace. Half of the women respondents indicated that it was “possible but difficult” or “worse” as characterizing their experience introducing changes in the workplace. In contrast, nearly 70 percent of the men considered it “very” or “fairly easy” to apply their KSAs. A number of factors may explain these differences. Professional women may have been dis￾persed through many organizations, which are in large part male-dominated in Africa, rather than concentrated in a few, where mutual support for change could have been built. In fact, women GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 10 Section I: Findings reported less support from colleagues and fewer resources to tap to help introduce changes. Per￾haps their influence levels were too low, or their career paths were blocked by promotion obsta￾cles that limited their abilities to induce change. It should be noted that women reported exten￾sive sharing of their knowledge among colleagues at levels that matched those for men. Al￾though the precise factors are not known that explain this finding, USAID program planners and other donors will want to design effective post-training support activities to ensure that women can bring changes in institutions more easily. Despite the obstacles encountered, women reported that the institutions in which they worked did change as a result of training. There were no gender differences in the extent to which there were “differences” at the institutions when participants succeeded in introducing changes. While quantitative survey results showed that women reported fewer achievements than their male counterparts, not surprising given the findings discuss above, it was apparent from anecdo￾tal information that women found ways to overcome these impediments to excel in ways uni￾maginable before training. Participants in general cited “increased self-confidence” frequently as a major by-product from their U.S. training experience, but these text citations could not be dis￾aggregated by sex to discover whether there were any gender differences in reports of “self￾confidence” increasing. It was clear, however, from dozens of interviews and participant meet￾ings that confidence levels among women graduates were extremely high. The unknown is whether they were so self-assured prior to their training, or whether other aspects of life in the United States affected their professional development more than the academic skills acquired. When designing such programs, sponsors and planners of long-term training should consider the constraints to women in applying their knowledge and skills to achieve their objectives when designing long-term programs. To what extent could impact from women participants have been even more remarkable had USAID, its local partners, and the African institutions been more pro￾active in ensuring, through follow-on activities, a supportive organizational context? Finding 11: No correlation could be found regarding impact and the frequency with which participants returned to their original workplace. Training assessment methodology for years presumed that an important factor to consider in measuring impact is the rate with which participants returned to their host institutions. No such correlation was found in this assessment. Participants reporting impact often worked in different institutions or even worked out of their country for periods. This having been said, nearly half (49.4 percent) of the participants surveyed did return to work at the same institution where they were employed prior to going to the United States. This ap￾pears to be an impressively high number, given the ups and downs of many African institutions over the years and the political instability that thwarted the return of many participants (to Ethio￾pia, Uganda, Ghana, Liberia) for years. The percentage includes recent returnees as well as re￾tired alumni. Twenty-five percent of the respondents stated that they worked for a different or￾GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 11 Section I: Findings ganization, with 17 percent indicating the question was not applicable or that they were not working. For those who migrated to another employer, most worked for only one or two organizations. Although these averages are deceptive since they include recent returnees as well as retired par￾ticipants, the data clearly supports a view that the ATLAS/AFGRAD participants remained in relatively few institutions. A normal career path would show at least two or three institutional changes, placing the participants well within the range of what would be considered a stable work environment. Since institutional change requires a minimum of work stability, it appears as if that prerequisite for change was easily met. On the other hand, given the dearth of institutional choice and employment opportunities in all African countries, the lack of mobility is not surpris￾ing. Not to be overlooked in this area is that people not returning to their sponsoring organization may not constitute a total “loss.” It is, of course, a short-term problem for the losing organiza￾tion, but it may not be as disappointing when one takes a long-term view and sees that the par￾ticipants may have made even greater and more widespread impact in other positions or even countries. Finding 12: Participants returned to their home countries after their U.S. training when conditions permitted. There is no significant evidence that long-term U.S. train￾ing under these sponsored programs contributed to any brain drain of African human resources. This study was not charged with verifying AAI’s claim that an average of between 85 and 90 percent of participants from all programs over the years returned to their country, which is an impressively high rate of participant return for any long-term U.S. training program. However, had the non-return rate been high, the assessment team would have confronted far more difficul￾ties locating participants in the seven countries visited, and through the survey sample. There would have been widespread complaints that nationals trained under the program disappeared overseas and never returned to build the targeted institution. The team heard very few such criti￾cisms. The problems in finding participants was due far more to poor contact information than to their absence from their home country. Moreover, the fact that a participant resides overseas at the time of this assessment in no way implies lack of impact or even that the person did not return to the home country sometime after training. The issue of rates of return is more complex than is generally thought and requires careful definition of terms used. The fact that the two programs stretched over such a long period makes the rate of return far more credible as a measure of re￾turn (not a measure of impact) since it flattens out the fluctuating security and economic condi￾tions of African countries over decades. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 12 Section I: Findings Finding 13: ATLAS/AFGRAD participants were well-advanced in their careers, making significant contributions to development. Since participants were selected into ATLAS/AFGRAD generally after some work experience and in collaboration with their employing institution, they were mature professionals even before starting a graduate education in the United States. Not only does this aspect of the program re￾duce the “brain-drain” threat that is characteristic of programs attracting younger university stu￾dents to U.S. graduate programs, but it promotes closer linkages between the academic program the participant undertakes and the development needs of the sending institution or country. The more important correlation, not substantiated in this assessment but in others, is between likeli￾hood of a participant returning with degree of donor collaboration with the sending institution (and the participant) in regards to needs assessment, selection of participant, design of program, and staffing plans to hold positions open during training. The fact is that all participants surveyed and interviewed, including those recently returned, were well advanced in their career paths and able to make significant contributions in their professional area. The case study on the following page illustrates well several of the findings above, in particular those addressing the non-technical impact from the U.S. training experience. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 13 Section I: Findings Impact Example 1 How Non-Technical Benefits of Training Can Yield the Greatest Impact Charles Ratsifaridana (AFGRAD), Nova Knits Deputy Director General Albain Rarivoson (AFGRAD), Nova Knits Manager, Production and Quality Control Madagascar Background Charles Ratsifaridana returned in 1980 with a PhD in Physics. He was one of six to start the Physics Department at the Ecole Normale Superiore. In 1986 he went back to the U.S. with the Humphrey Pro￾gram and studied Information Systems and Management at the University of Pittsburg. He returned and worked as a consultant for many companies in 1987-88. In 1991, Nova Knits asked Charles to open and manage a new cashmere sweater factory. The owner wanted a senior manager with common sense, a background in computer management, and U.S. experi￾ence. At the time Charles started working at the factory, there were 400 employees. By 2000, it counted 7,000 employees, which was scaled back to 4,000 with the 2001 crisis. Nova Knits then hired Albain Rarivoson, an AFGRAD participant who had studied Mathematics in the U.S. Charles believed that “if he can do mathematics, he can do anything,” and put him in charge of environmental standards, and compliance. Charles said that they both understand each other and their background (from U.S. train￾ing) and know what to expect of each other. Right now business is expanding and Albain is in charge of middle management training. Albain loves teaching and will be teaching topology at the university in the town where the new factory will be lo￾cated. Impact Self-confidence. Charles stated that he never would have applied for a job in a field with which he was unfamiliar had he not gone to the U.S. and developed his self-confidence. He knew Nova Knits needed someone with computer management skills and simply told the Director, “I am the one you need for this job.” Values. Individual values and drive are more important than degree level or even field of study. Even though he had a degree others envied, he chose to do work that was more fulfilling. Charles said that it would have been difficult to build this mind-set had he not studied at U.S. universities. English Fluency. Having English capability was critical in Madagascar since textile buyers from over￾seas speak little French and prefer dealing with Charles and Albain. Continuing Education. Charles also feels that they “can only develop the country through education.” He is working with two associations (GEFB and MEMA) of English-speaking companies to build the partnerships. Through this association, he can talk to individual companies to try to do activities that are sustainable. Charles also has an observatory on top of a mountain with a 15 meter tower and 2 tele￾scopes. He has invited approximately 1,000 students in the field use it. Human Resources. Nova Knits invests in training, and Charles accepts that some trainees have left the company. His perspective is that although they have gone to other companies, they still have contrib￾uted to development since they have stayed in Madagascar. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 14 Section I: Findings C. Laying the Groundwork: Analyzing Impact Using the Modified Framework Surveys of returned participants reported high levels of impact. It is clear that participants them￾selves believed they made major contributions that were the result of their academic training in the United States. One data table alone cannot demonstrate that significant impact flowed from USAID’s investment in U.S. degree training. But taken together, as shown in this section, the data already lead to the reasonable finding that verifiable impact occurred and was significant. 1. Acquisition of Knowledge, Skills, and Attitudes Before impact can occur, participants must acquire the knowledge, skills, and attitudes from the training program. Did the ATLAS/AFGRAD participants meet this first test for impact to occur? Of the weighted estimate of the total number of participants shown in the table for Question 36, 99.7 percent of those surveyed reported that they acquired specific KSAs during training. Some might claim that it would have been surprising had an individual not acquired some useful skills after several years in the United States. Although this view is legitimate, the point here is that a trainee is not likely to introduce substantive changes in the workplace, sector of activity, or nationally without having learned something from the academic program. In other words, one logi￾cally must pass from Kirkpatrick’s Level Two (Acquisition) to Level Three (Application) to move closer to impact beyond the individual. If the person does end up contributing significantly without having acquired the skills, which is possible, then the attribution between training and result is brought into question. In that case, USAID’s investment did not produce the anticipated result, but perhaps another donor’s overseas training produced the change the participant claimed, or another factor is present that is unrelated to the training (that is, improved economic stability that enabled a person to succeed). Table 2. Acquisition of Knowledge, Skills, and Attitudes during Training Question 36: Did you acquire any specific knowl￾edge, skills, or new attitudes from your academic program in the United States? By Gender Yes % No % Total % # Male 99.6 0.4 100 1217 SE 0.2 0.2 Female 100.0 0.0 100 654 SE 0.0 0.0 Total 99.7 0.3 100 1871 SE 0.1 0.1 (Non-responses: 50 (2.6%) out of 1921) SE = Standard of Error Self-perceptions of impact do not verify that it occurred. Alone, a participant’s own assessment is subjective, tainted by numerous factors, such as pride, false sense of importance or influence, and the ever-present hope that a positive spin on the return on USAID’s investment in training might lead to future training opportunities (either for the participant or compatriots). However, when answers to “yes/no” or multiple choice questions are grounded by participants’ examples, the credibility of the statistics cited is augmented. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 15 Section I: Findings In response to survey Question 29, 98.9 percent reported that the contri￾bution made by their U.S. training to their professional development was “very important or important,” with 82.0 percent opting for the highest of these two categories. To justify their responses, partici￾pants were able to cite plausible, verifiable examples of the knowl￾edge, skills, and new attitudes that they claimed to have acquired when answering “yes” to Question 36. Had they left the spaces empty, entered examples lacking a logical develop￾ment impact (such as, “I learned a lot about the world’s superpower” or “I learned how to drive”) or entered vague examples (such as, “enhanced my knowledge”), the credibility of the answer to Question 36 would be in doubt. But even discounting a certain number of “false” affirmations, the number of returned participants who claimed to have acquired skills, and given acceptable examples, is impressive. Table 3. Contribution of Academic Training to Professional Development Question 29: In relation to my professional development, I believe that the contribution that my ATLAS/AFGRAD [training] made to my professional development was: By Gender Very Im￾portant % Important % Not Very Important % Total % # Male 85.2 13.3 1.5 100 1249 SE 0.4 0.9 0.3 Female 76.0 24.0 0.0 100 653 SE 0.8 1.5 -- Total 82.0 17.0 1.10 100 1902 SE 0.4 0.8 0.2 (Non-responses: 19 [1%] out of 1921) SE = Relative Standard of Error The principal KSAs that participants claimed they acquired during training and wrote onto their questionnaires are displayed in the text box on the following page. What is illuminating in these free choices by surveyed participants is the consistent emphasis on a variety of skills that support research methodology over the technical knowledge acquired. In other words, participants considered their research style more noteworthy than their improved technical competency, which they selected as second in importance. Agricultural specialists could easily claim to have learned, for instance, soil science or plant pathology; but without learning how to carefully design and conduct research and analyze the results, fewer results would be forthcoming in terms of applied solutions to increasing yields. Could it be the case that increasing technical skills, however critical to human capacity growth, is an insufficient compo￾nent to achieving impact? GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 16 Section I: Findings Impact Example 2 Examples of KSAs Acquired during Academic Program (in priority order according to frequency of mention) % The most important KSAs % The 2nd most important KSAs 15 Designing/conducting/analyzing scientific research 15 Designing/conducting/analyzing scientific research 12 Technical/scientific method/tools 12 Attitude towards work 7 Problem analysis, learning & critical thinking 9 Management 6 Leadership 8 Technical/scientific method/tools 5 Attitude towards work 8 Computers 5 Computers 7 Problem analysis, learning & critical thinking Question 37: If [you acquired any specific knowledge, skills or new attitudes (“KSA”) from your academic program, please indicate examples of the THREE most important [KSAs] that you acquired.... Participants wrote 150 open-ended text answers for the “most important” option (the left column above), 115 for the 2nd column and 117 for the 3rd, which is not shown. Text entries for the 3rd category followed the same trends as indicated elsewhere. The Six Most Important KSAs Acquired (averaging the two columns above) % Designing/conducting/analyzing scientific research 15 Technical/scientific methods/tools 12 Attitude towards work 9 Problem Analysis, learning & critical thinking 7 Computers 6 Management* 6 * “Management” and “leadership” made the top rankings on only one list above, but the former received slightly more mentions than “leadership.” The others appeared on both lists. (The complete list is included in the Annexes.) These quantitative findings are amply buttressed by anecdotal information gathered during the site visits. Time and time again participants emphasized the improvements they brought home in research techniques or teaching methodologies over the technical knowledge they gained in graduate school. Another discovery from the above participant survey is the high importance placed on work atti￾tudes. It is unlikely that prior to arriving in the United States, any participant would have imag￾ined that a major fruit of their training program would be improved attitudes toward work. That it was ranked by respondents in third place is extraordinary—even more so when realizing that this training “outcome” was never anticipated or articulated by USAID or its partners. For years many gave lip service to the time-tested rationale for sending participants to the United States: to GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 17 Section I: Findings change attitudes, win friends, establish linkages, and so forth. But rarely were these factors inte￾grated into program planning for the U.S. “experience.” Few participants were placed in intern￾ships at U.S. companies, NGOs, or government agencies during their stay, although, in all likeli￾hood, many arranged work experiences themselves—on campus, after their degree, or elsewhere. That computers and management make the cut of the most important skills acquired is perhaps not surprising in one sense. Since few Africans coming to the United States prior to the 1990s had exposure to computers, they considered this an important asset upon return. In terms of management, that it made the list is in itself significant, since acquiring management skills was never a stated objective or focus of the two programs. Later programs in the 1980s and 1990s offered two-week management seminars to participants during academic breaks, but aside from these add-ons, little programming occurred (internships with organizations, NGOs, research facilities, and so forth) to build participant capacity to manage their professional work in a systematic way. 2. Application of Knowledge, Skills, and Attitudes Which of these acquired knowledge, skills, and attitude changes were applied by the participants in their institutions? To ascertain whether the training made a difference, which is the objective of the assessment, evaluators must consider whether the fruits of the training were applied (Kirkpatrick Level 3). Otherwise, the change obtained through training remains at the individual level—a distinction that has become critical in assessing the impact of the investment in human capacity improve￾ment. Ninety-five percent reported that they applied their new KSAs at the workplace “a great deal” or “a lot,” a high figure by any stan￾dard. Following the same approach to grounding affirmations, as was done for Level 2, the questionnaire probed which of the KSAs listed were applied. The examples once again largely confirm the credibil￾ity of the responses to Question 38. Again, participants clearly indi￾cated in the text box below that the skills they succeeded in using in the institutions were less related to their precise technical capacity and more linked to their managerial, research, teaching and learning skills. Table 4. Application of Knowledge, Skills, and Attitudes at the Work Place Question 38: If you answered “yes” to Question 36 above, how much of the knowledge, skills, or new attitudes ... did you apply in your work? By Gender A great deal % A lot % Some % A little % No Opinion % Total % # Male 61.4 32.9 4.8 0.9 0.0 100 1245 SE 1.4 1.3 0.6 0.3 -- Female 64.3 32.3 2.6 0.0 0.0 100 653 SE 1.9 1.8 0.6 -- 0.3 Total 61.9 32.9 4.2 0.6 0.3 100 1898 SE 1.1 1.1 0.5 1.7 1.2 (Non-responses: 23 (1.2%) out of 1921) SE = Standard of Error GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 18 Section I: Findings During meetings in-country, participants often spoke about applying strategic planning and monitoring skills they learned during training to their work at home to accomplish results. One participant wrote: The greater professionalism, positive attitude, better understanding of dif￾ferent issues, people and cultures, better teamwork ... all enhanced per￾formance and productivity. How difficult was it for participants to apply their new knowledge or to impart new attitudes on colleagues? Answers to Question 41 reveal that “possible but difficult” was selected the most by all participants to charac￾terize the level of ease in introducing changes based on training once returned. But “very easy” and “fairly easy” together represent 62 percent of the estimated total of par￾Table 5. Degree of Difficulty in Applying KSAs at the Work Place Question 41: Please indicate how difficult or easy it was to apply your knowl￾edge, skills, and new attitudes where you worked immediately after returning. Very Easy % Fairly Easy % Possible but difficult % Very Difficult % Impossible & Total % # Total 34.0 30.4 31.6 2.6 1.3 100 1645 SE 1.8 1.6 1.7 0.4 0.3 (Non-responses: 276 (15.4%) out of 1921) SE = Relative Standard of Error Impact Example 3 Examples of Skills Applied at the Workplace (in priority order according to frequency of mention) % Knowledge, Skills and Attitudes Applied 49 All KSAs listed in Q37 8 Managerial 8 Research 5 Teaching and Learning 5 Technical skills 5 Computers 20 Other Question 39: If you answered “a lot” or “a great deal” [to Question 36 about applying KSA in your work], please indicate which ones that you applied. There were 151 open-ended text answers to this question, of which 74 (49 percent) were “all of the above” or variations of that choice. To get an idea of the relative importance respondents gave to those skills, con￾sult the table showing answers to Q37. The remaining 77 (51 percent) selected the skills shown in the table. (The complete list is included in the Annexes.) GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 19 Section I: Findings ticipants. Participants encountered significant difficulties, it would appear, yet overall succeeded in introducing change.9 Participants recount many examples of overcoming difficulties after returning to their institu￾tions. They have acquired innovative techniques to succeed often against considerable odds. In education, often the students themselves support changes introduced by returning participants, as shown in the example below: At the beginning there was a training culture clash between the French and the U.S. education traditions. This was made worse by problems with diploma equivalency. But, our U.S. graduates are gradually winning the battle because the students have demonstrated that they prefer the pragmatic and participa￾tory way of teaching. And because we are dealing with teacher training, the approach will be more easily and quickly transferred in the system. Rene Rassouanaivo, Director Ecole Normale Superieure, Madagascar Another factor in paving the way to introducing changes in the home institution is the number of U.S. graduates working together. A critical mass of technical experts can make a difference in an institution, although as discussed elsewhere in this report, alone it is not sufficient to obtain im￾pact over time. To what extent did the simple fact that, for instance, half of the professors in the faculty of education at a given university over a several decades shared the common language of a U.S. graduate education, explain why change occurred? That a small group of “change agents” seemed to work together to produce a supportive work culture seemed was perhaps the unrecog￾nized factor that made change happen. 3. Measuring Changes in Institutional Output The highest level of change associated with Kirkpatrick’s evaluation levels (unmodified) is Re￾sults (Level 4, also referred to as “Outcomes” or “Change”). This assessment in essence ex￾panded the evaluation framework, as explained in Volume II, Annex C, to accommodate the pos￾sibility of measuring impact beyond the institution (for example, in or across a sector, nationally, regionally, and internationally). But before considering impact at that level, the team sought to first determine whether institutional change occurred and could be attributed to USAID’s in￾vestments. One way to query whether Level 4 changes occurred is to avoid the use of one or two terms such as “result” or “change” when posing the question. African organizations often do not measure their output or track results, so that a respondent might dismiss impact that may have resulted from changes introduced due to conceiving the question too narrowly. What evaluators want to 9 For examples of the reasons the small number of participants (5 percent) indicated no success in introducing change at their workplace, consult the discussion under “Gender” where the participant selections are shown and male and female responses are compared. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 20 Section I: Findings know is whether there has been any difference in the institution’s output, performance, produc￾tivity, or quality—anything noticeable that could be attributed to the training received and KSAs applied. Of the participants who answered survey Question 47,10 96 percent affirmed that there was a change in terms of their institution’s performance, productivity, or impact—in other words, something positive resulted from their efforts. Following this assessment’s commitment to grounding such “yes/no” questions, participants were asked in Question 48 to provide concrete examples demonstrating that institutional change occurred, some of which are shown in Impact Example 4 below. Impact Example 4 Examples of Institutional Changes Inspired by U.S. Training (Question 48: Please give one or two concrete examples below.) 1. Government agency now publishes articles on agricultural productivity on a regular basis 7. Research quality improvements led to better vaccines 2. Reduction from 2 wks to 1 day the time needed 8. More students get involved in research now to complete a balance sheet 9. All computers are connected full time now 3. Built a factory employing 500 people which led to creation of 1,500 indirect jobs 10. More effective rural education programs led to more women accepting vaccinations 4. Started a customs laboratory after training to test imported drugs and food that has saved lives and generated income 11. Entire university curriculum was changed to more flexible modular course-credit system 5. Rural banking expanded due to changes introduced 12. Entire university curriculum was changed to more flexible modular course-credit system 6. 300% increase in profits in company 13. Cassava mealy bug was successfully con￾trolled in the 1989-1995 period 14. Learning increased in targeted departments due to improvements in teaching methods The high 96 percent affirmation that institutional change occurred, combined with the examples put forward by the participants captured in the table above, strongly suggest that measurable, significant changes were introduced at institutions that “made a difference.” A glance at the complete list of answers to this open-ended question (contained in the Annexes) underlines how extensive the changes appear to be. Of the 134 examples written by the 203 participants who completed questionnaires, roughly 75 percent were valid examples of institutional output changes. Twenty-five percent were either examples of management or technical interventions 10 Q47: If you were able to apply your knowledge, skills or new attitudes at work..., has there been any difference in your institution’s output (in terms of quality, quantity, and so forth), performance, productivity, or impact? In other words, did anything change? GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 21 Section I: Findings introduced at the institutions, such as “better teamwork” (instead of the output changes they might have produced), or disingenuous statements of personal accomplishments. The examples put forward are not objectively verifiable, to be sure. It would not be realistic to perform due diligence to determine whether each change occurred, to what extent it was attribut￾able to the participant’s training, and what the change brought in terms of impact to the institu￾tion, sector, nation, and so forth This line of inquiry would be far too exhaustive. Instead, the ex￾amples cited by participants in their words can be put to a test of plausibility: Are the examples believable? Is it possible to imagine them being introduced? When the survey answers to Question 48 are supplemented by the individual participant state￾ments describing their impact in their institutions (or beyond) that the assessment team collected in the seven countries visited, it becomes clearer that significant improvements in institutional performance likely occurred throughout the period. Many participants freely describe credible changes they witnessed resulting from interventions they linked to their training. The 96 percent affirmation should not be dismissed due to the built-in bias associated with self￾assessments. In fact, African employees, as well as expatriate organizational development spe￾cialists, are quick to blame rigid, old-fashioned institutions for preventing innovation and block￾ing changes introduced by those returning from overseas programs. Impact assessments often show high rates of KSA “acquisition” but lower rates of organizational change, citing predictable reasons such as lack of leadership at the top, professional jealousies, and resource constraints. But the institutional changes that appear to have been made with the help of ATLAS/AFGRAD returnees appear to be numerous, noteworthy, and credible. Both the number and quality of ex￾amples produced (from the survey and from site visits) bear witness to a high degree of positive impact in institutional output (Kirkpatrick Level 4). ities. Central Veterinary Laboratory, Mali The training has contributed to building the capacity of the Central Laboratory so that all the technical services which were run by expatriates are now held by the Malian graduates. Lab Staff stay in permanent contact and cooperation with U.S. univers D. Dissecting Impact at Higher Levels GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 22 This assessment differs from earlier ones in that it seeks to identify and quantify impact at levels beyond the institution. Since Donald Kirkpatrick published his four evaluation levels in 1958, the debate has raged among training and human resource specialists over the number of levels needed to measure the impact from training. Some prefer five, others six, and many find limitations in focusing on just four levels. Noting that Kirkpatrick’s framework was meant for North American private businesses, it should not be surprising that those assessing training im￾pact in a country’s agricultural sector, or in its trade policy formulation, would want to delve Section I: Findings country’s agricultural sector, or in its trade policy formulation, would want to delve beyond a particular institution. This assessment would not want to exclude possibly significant impact at higher levels, especially in view of the length of time between a participant’s return and the po￾tential for impact (which was, in some cases, over 30 years). Did impact occur within a sector, community, nationally, in a region, or internationally in a manner that could be linked to long-term training? These divisions are fluid when impact is con￾cerned. For example, an institution that developed and commercialized an improved seed variety could very well impact the agricultural sector, the nation in terms of agricultural production, and quite possibly the region, as neighboring countries learn and perhaps use the promising innova￾tion. International impact might even unfold, as researchers and agri-business conglomerates abroad hear about a success in far-off Mali or Malawi. With this reality as a backdrop, this sec￾tion takes a brief look at whether any impact was noted at higher levels. Participants were asked in survey Question 49 to indicate at what level the change you describe [in Question 48] took place. Following are the levels participants were asked to select (from low￾est to highest level). Table 6. Location of Impact Male Female Both Level of Impact Selected 1 1 1 Within the Institution where you worked 4 2 4 In the Sector targeted by the institution where you worked 2 4 3 In the Community 3 3 2 Nationally 5 5 5 Regionally (such as West Africa) 6 6 6 Internationally Predictably, institutions were selected first by all participants together, then nationally, followed by the community and sector. Indicating national impact at such a high level, before sector, may reveal that in the eyes of most, any positive change in a sector would necessarily have a national impact in most African countries. 1. At the Sectoral Level Two questions merit consideration under the rubric “sectoral.” First, were there differences in impact registered according to sectors? In other words, did graduates in health have greater im￾pact than those in agriculture or education? Second, what impact occurred at the sector level re￾gardless of the field (agriculture, health, and so forth)? Both questions are treated in this section. The survey instrument included a variable that enabled the data to be disaggregated by the fol￾lowing sectors: Agriculture, Health, Education, Financial, Transportation, Management, Admini￾GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 23 Section I: Findings stration, and Other. Participants were asked to indicate the sector in which they worked prior to training, and again afterwards. The combination of those two was used to deduce the sector in which the participant worked. The principal point of interest in considering sectors is to determine whether participants in one sector tended to report more or less impact than those from others. If so, what is the explanation, and are there lessons to learn from this experience? There may be trends or tendencies to weigh for future programming. In terms of application of skills, as shown in the fol￾lowing table, the health, education, and financial sectors were roughly the same in the column (“a great deal”). Agriculture, on the other hand, stood out from the others in terms of its lower rating in that column. Did participants in agricul￾ture indicate more diffi￾culty in applying their KSAs? Not necessarily, according to the answers to survey Question 41 (not shown) (how difficult or easy was it to apply your knowledge...?). Data indi￾cated that although over half of the agriculture respondents stated it was “fairly easy” (but not “very easy”) to apply KSAs, double the number of education participants (57 percent) selected “very easy” compared to those in agriculture. Health graduates had the hardest time, according to the data (51 percent choosing “possible but difficult”). Interestingly, a whopping 81 percent of those in the financial sector reported difficulties in applying their knowledge in their sector. (See Volume II, Annexes for the table for Q41.) Table 7. Application of KSAs by Professional Field Question 38: If you answered “yes” to Question 36 above, how much of the knowledge, skills, or new attitudes... did you apply in your work? By Sector A great deal % A lot % Some % A little % No Opinion % Total % # Agriculture 37.8 56.4 5.8 0.0 0.0 100 362 SE 2.6 2.6 1.2 -- -- Health 73.7 26.3 0.0 0.0 0.0 100 167 SE 3.0 3.0 -- -- -- Education 78.9 17.6 2.6 0.0 0.9 100 546 SE 1.7 1.6 0.7 -- 0.4 Financial 69.9 14.7 8.3 7.1 0.0 100 156 SE 4.1 3.3 2.6 -- 0.4 Transportation 21.2 21.2 57.6 0.0 0.0 100 33 SE 7.1 7.1 8.6 -- -- Management/admin 75.8 24.2 0.0 0.0 0.0 100 62 SE 3.7 3.7 -- -- -- Other 64.5 32.2 3.2 0.0 0.0 100 301 Non-responses: 174 (9.7%) out of 1921) SE=Standard of Error How supportive were those who supervised work—another triangulation to discover the level of difficulty of generating impact among sectors? Data from survey Question 43 (How supportive were those people who supervised your work when you applied your newly acquired knowl￾edge?) substantiate the findings above: by far, people in education received more support from supervisors (65 percent “very supportive”) than those in agriculture (21.5 percent) and health (21.9 percent). The financial sector fared worse again, with only 14.4 percent finding their su￾pervisors “very supportive.” GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 24 Section I: Findings Impact Example 5 How Impact on a Sector Can Occur from Afar Dr. Eddie Mukasa Mugerwa Kampala, Uganda Personal Statement from a Participant Note: The hand-written statement below articulates clearly that sectoral impact can be sustainable despite political unrest at home. It illustrates that “returning” to country can be an inappropriate indicator of impact. The author shows that even during 22 years of absence, he continued to have an impact in his sector and on his country from afar. When conditions improved, he returned. Dr. Mukasa-Mugerwa received a doctorate in animal science from Colorado State University in 1974. I was selected as an AFGRAD Fellow at a time when Uganda felt the need for better trained manpower to improve animal agriculture. I accomplished my objective and started to teach at the university, the idea be￾ing to multiply the knowledge acquired through higher level training and research. The training received was relevant, and … all went well until political upheaval broke out. This constraint made it impossible to con￾tinue usefully and I left. However, I continued to make useful contributions since in my new position at the International Livestock Research Institute, [in Ethiopia] where I worked for 22 years as a researcher, I produced over 100 manu￾scripts, many of which are used by students and staff at my parent institution, Makerere University. Since my return 5 years back, I have continued as a researcher, consultant, and farmer, and even received the Cochran Award, a small contribution to animal agriculture through research and training. This is much appreciated by my family and numerous students I have supervised in Uganda and outside. In terms of contributions outside the workplace (Q45), differences in institutional output (Q47), and noteworthy professional discoveries or contributions (Q51), in contrast to the above observa￾tions, there were no major differences between agriculture, health, and education (where an aver￾age of 85 percent reported the highest levels in each table). The financial sector, however, again varied significantly from the other three with regards to Q51: fully 87.2 percent reported no achievement or contribution made. Impressions gleaned from interviews and participant meetings did not support the indications from the quantitative data that there was less impact in the agriculture sector than others. Many signifi￾cant examples of impact were heard in the sector, as with the others. But the data may be pointing to institutional differences, especially when comparing agriculture and education, in terms of pro￾viding a supportive context within which organizations can evolve and produce results. 2. At the Community Level The assessment team included “community” as an impact level to consider—perhaps without sufficiently reflecting on its meaning. First, it does not fit the neat hierarchy established by GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 25 Section I: Findings Kirkpatrick, and expanded by others, whereby impact is monitored at the lowest to highest levels by tracing a trainee’s degree of satisfaction, KSA acquisition, application of KSAs in the organ￾izational unit, and resulting institutional output changes. The community challenges the notion of impact flowing consecutively, chronologically, or even measurably in a work context. To some, impact in the community would presume unanticipated impact from training, that is, changes introduced outside the workplace by socially conscious participants. Yet unanticipated impact, however desirable, should not, according to some, be included in the scope of an assessment of the value derived from investments in long-term training. Another dilemma in assessing community impact is that the “community” is undefined. At a minimum, the terms institution, national, regional, and international have recognizable bounda￾ries, however fluid. Community remains far more elusive: it can be a physical place, such as a village, yet at the same time, can refer to a localized network of people, such as “university stu￾dents” or “agricultural workers.” It can easily cover a nation yet be decentralized (women’s co￾operative movement) or international without being local (Internet-based interest groups whose members do not know each other). The survey instrument sought to isolate community impact without defining the term. The table at the beginning of this section indicated that participants ranked “community” highly in terms of their perceived impact. But there is no way to separate out impact derived from application of KSAs at the workplace, from impact on community, from a participant’s contributions outside work. The list of open-ended responses (see Volume III, Annex D) contains examples of community impact cited by the participant. Individual “Impact Statements” written by participants during site-country meetings provided anecdotal examples of substantial impact at the community level, such as the one that follows from Namibia. The participant emphasized that his changes were the result of teamwork and that he could not take credit alone. The impact was team-based and occurred at the local level, and somewhat at the national level. We formulated a peri-urban land use policy to balance de￾velopment and the environment by identifying sites for urban, recreation, and industrial use. The change can be measured by the number of applications re￾ceived from municipalities that were rejected because they did not conform to the Environmental Impact Assessment policy we established. As a result of our efforts, [a major community in an ecologically sensitive area] formed a “Dune-Belt Management Committee” that monitors the site. The Namibian case above underscores the fluid nature of impact affecting, in this case, the local level and to a lesser extent, the national level through policy change. Community changes most likely occurred in many instances that were undiscovered by the assessment team. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 26 Section I: Findings 3. At the National Level As discussed previously, participant rankings of the level of impact demonstrate that national impact was thought to be frequently attained by participant contributions, second only to institu￾tional impact. The open-ended question (Q48) asked participants to provide examples of impact at the levels they cited. The list of responses (included in the Volume III, Annex D) presents a wide array of concrete examples of changes at different levels. A few illustrative examples given by the participants are: • Through my training, I represented my institution at a World-Bank financed national project; • Developed teaching materials for the national curriculum panel; • My institution advised the government on agricultural matters and food production; • I applied skills I learned within the institution that I worked and helped to improve the quality of services rendered to the business community in bilateral trade between Mo￾zambique and Kenya; • Collective and constant teaching about the consequence of unprotected sex have brought about behavioral change in Uganda youth, which has led to a significant decline in HIV/AIDS prevalence; • In the Environment Ministry, we were able to approve major laws that will enable the country to apply responsible policies of development. 4. At Regional and International Levels Participants selected “regional” impact next to last in responding to survey Question 49 and “in￾ternational” last. However, ample examples were given in text responses that indicate both “re￾gional” and “international” is, of course, not clear in many instances, nor relevant. A Ugandan educator traveling to Namibia to share new teaching methodologies she developed for rural pri￾mary school teachers encounters North American educators attending the conference interested in her discoveries. Is this regional or international impact, or does it make any difference? To attempt to capture higher-level regional and international impact emanating from an institu￾tion or a participant, the assessment developed the Internet Impact Search (see Volume II, Annex G). Since that search concentrated on ferreting out impact at the regional or international levels, its findings, summarized below, should be considered in this section. • Fifty-one percent of the randomly selected names were found on the Internet where some degree of impact could be determined. • A large number of the 51 matches hold or have held positions of high responsibility: o 3 Cabinet Ministers o 1 elected member of Parliament o 6 Directors or above in governments, donor agencies or NGOs GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 27 Section I: Findings Impact Example 6 How Impact Saved Lives and Improved Health Mrs. Dinah Brandful Ghana Customs Excise and Preventive Service After 5 years as a quality control officer for a food processing company, I went to the United States under the AFGRAD scholarship program to further my education by reading for a master’s degree in Food Science at Michi￾gan State. Upon my return in 1983 I continued to work for the company but joined the government’s Customs Excise and Preventive Services (CEPS) department in 1985. This unit was an autonomous body with its own laboratory to do product analysis and identification for imports and exports. I was soon made an assistant to the head of the labo￾ratory department because of my academic qualification and experience. With the help of the head of Department, we designed and structured the whole laboratory department. Today we have staff positioned at all our border exit and entry points to engage in physical inspection of products and referrals of products to the laboratory. Some of the products we analyze and identify include Textiles, Food, Alcohol, Drugs, and so forth Impact Currently we have 36 employees including supportive staff. Our laboratory is also regarded as a centre of excel￾lence in the West African sub region. Due to our reputation, Ghana hosted the first conference on customs laborato￾ries for Africa in 1995 to help other countries set up similar facilities and to harmonize practices. As head of the laboratory, one of our major tasks is raising revenue for the government with correct analysis and product identification that helps determine the correct duties and taxes that government should collect. A second major impact is on the lives and wellbeing of Ghanaians. We regularly analyze and seize or ban unsafe pharmaceuticals entering from India or Nigeria. We do not know exactly how many lives our interventions have saved. We are also moving towards ISO-9000 certification under a program funded by the World Bank. My study in the US gave me a different positive attitude and equipped me with different skills and ethics such as hands on approach, confidence and assertiveness as well as insisting and fighting for my rights as a citizen. o 2 elected officers of African regional organizations o 1 founder of a national NGO • A high number of the 51 matches revealed demonstrated impact as shown below: o 20 authored or co-authored papers or reports, most with international exposure o 8 authored or co-authored books in their areas of expertise (of which one also be￾came a popular novelist) o 6 delivered presentations or papers at international conferences o 2 delivered presentations or papers at Africa regional conferences o 2 delivered presentations or papers at national conferences o 3 have won national or international awards for specific accomplishments and/or life work GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 28 Section I: Findings o 1 received a national award from a women’s organization mentored by the par￾ticipant’s NGO Details about these impressive initial discoveries of impact are not known due the inability to expand the search beyond initial resources allocated. In many cases, according to the re￾searcher’s report (included in the Annexes), participants had multiple “hits”—often as many as several dozens—reflecting a variety of international arenas where they were known. A sampling of some notable hits that popped up in the Internet search includes: • Miriam Khamadi Were—an accomplished Kenyan novelist and teacher who later became a medical doctor, university professor, head of a UN office in neighboring Ethiopia, and Chair of the African Medical Research Foundation and of the Kenyan National Aids Control Council. • Mulatu Wubneh—an Ethiopian professor of planning and development who has taught at several universities in the US, and who spent an 18-month stint helping set up the Africa Capacity Building Foundation, a joint World Bank, UNDP, and African Development Bank initiative. • Linda De Vries—a South African university professor who also started a women’s em￾powerment investment group, and chairs the Western Cape Gambling and Racing Board. • Chiekh Ibrahim Fall—a Senegalese international civil servant who has served as the Vice President and Corporate Secretary of the World Bank, as well as holding a string of high￾level posts at the African Development Bank. Evidence of participants holding positions of international influence was provided by AAI in documents submitted to the team. Some illustrative examples of these alumni and their positions are included in section F.1 of this Volume (Table 17). E. Considering Impact by Other Variables Several other variables proved useful in understanding the characteristics of impact. In addition to obvious variables, such as gender and degree, that would lend light on impact and provide les￾sons learned, the assessment included Africa’s linguistic divisions in its analysis. These variables are addressed in this section. 1. The Gender Factor Some of the questions that would be instructive to answer to determine the gender factor in im￾pact might be: • Did women participants acquire and apply their KSAs at rates different from men? • Did women have more or less difficulty in applying their knowledge and skills at their workplaces? GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 29 Section I: Findings • If yes, why, and what were those difficulties? • Did women report more or less institutional change than men? • How do the achievements or contributions of women compare to men? • Is there any difference in career advancement? In terms of acquisition and application of knowl￾edge and skills, both of which bear directly on the likelihood of impact, were there gender dif￾ferences in the data? The two principal survey questions turned up no major differences noted between male and female participants, according to the data presented in the table for survey Question 36. Survey Question 38 (“how much of the knowledge ... did you apply”) allowed for multiple responses; women actually responded a bit higher but the differences are not significant statistically. At the point where women returnees tried to implement changes in their institutions, they clearly reported more difficulty than men. When the data from first two columns in the table for survey Question 41 are combined, 69.2 percent of men compared to only 50.4 percent of women considered “very” or “fairly” easy their tasks. Noteworthy is that nearly the same number of women (49.6 percent) chose “possible but difficult” or worse (“very difficult” and “impossible”) as characterizing their experience in intro￾ducing changes in their workplaces. Only 30.8 percent of men indicated this level of difficulty. Table 8. Gender Comparison of the Acquisition of KSAs during Training Question 36: Did you acquire any specific knowl￾edge, skills, or new attitudes from your academic program in the United States? By Gender Yes % No % Total % # Male 99.6 0.4 100 1217 SE 0.2 0.2 Female 100.0 0.0 100 654 SE 0.0 0.0 Total 99.7 0.3 100 1871 SE 0.1 0.1 (Non-responses: 50 [2.6]%) out of 1921) For hints as to why women were less suc￾cessful in using their newly acquired knowledge, skills, and attitudes, partici￾pants were asked to select from a list of seven reasons, re￾peated below. The following table shows the different ways men and women made their selec￾tions. Table 9. Gender Comparison of the Degree of Difficulty in Applying KSAs Question 41: Please indicate how difficult or easy it was to apply your knowledge, skills, and new attitudes where you worked immediately after returning. By Gender Very Easy % Fairly Easy % Possible but diffi￾cult % Very Difficult % Impossible % Total % # Male 35.4 36.4 26.2 2.0 0.0 100 1013 SE 2.4 2.4 1.9 0.4 -- Female 31.8 20.7 40.3 3.6 3.5 100 632 SE 2.7 2.0 3.3 0.8 0.8 Both sexes 34.0 30.4 31.6 2.6 1.3 100 1645 SE 1.8 1.7 1.7 0.5 0.3 (Non-responses: 276 (14.4%) out of 1921) SE=Standard of Error GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 30 Section I: Findings Table 10. Participant Reasons for Not Applying KSAs at Work Male Female Reasons for Not Applying KSAs at Work 6 1 I do not have the support of my colleagues. 5 1 I do not have the necessary equipment or resources. 2 3 My present work does not require the skills I learned in my ATLAS/AFGRAD program. 4 3 I do not have the authority to put my training into practice. 3 5 There was no work in my area of training or study. 1 5 I do not have the support of my superiors or supervisors. Note: Rankings in the “Female” column that are repeated (1, 1, 3, 3, 5, 5) indicate that the same percent of respondents selected those reasons. With the choices ranked in priority order by female respondents, the differences between genders become clear: an almost inverse order of reasons explaining constraints to applying KSAs at the workplace. Given the small percentage of the weighted total that responded to this question (5 percent), a few observations that could be made are that women (a) may have encountered more resistance from colleagues, (b) had less access to resources, or (c) were not in positions of suffi￾cient influence to implement changes. In terms of the degree to which women shared their KSAs with others (Q42), there were no differences in the data by gender. These observations are instructive for donors that seek ways to support impact in the post￾training environment. It may be that women returnees need targeted interventions from donor agencies to ensure that the fruits of long-term training are cultivated in a supportive organiza￾tional setting. Despite the higher level of difficulties women reported in their workplaces, a high percentage of women and men noticed differences in their institution’s output (Q47). Overall, with virtually no difference by gender, 96 percent of the weighted participant total who indicated success in apply￾ing their KSAs indicated changes in their institutions. As shown in the discussion in part C.3 of this Volume, participants grounded their answers with specific and acceptable examples of insti￾tutional change. Although the open-ended answers were not disaggregated by gender, quality examples were found throughout. If institutional output did change, in spite of some higher levels of difficulty encountered by women, were there differences in career advancement or professional contribution between the genders? GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 31 Section I: Findings The table for Question 51 shows a fairly signifi￾cant difference, with double the number of women indicating no achievement than men (35.9 percent against 18.9 percent). Many women attended the participant meetings held in the seven countries visited and were inter￾viewed by the assessment team. Their impact statements, presentations, and testimonies clearly demonstrated that they recorded highly signifi￾cant impact related to their training. Despite their encountering more difficulty in applying the benefits of their training, many found ways to overcome these impediments to excel in ways unimaginable before training. To what extent, however, could their impact have been even more remarkable had USAID, its local partners, and the African institutions been more proactive in ensur￾ing, through follow-on activities, a supportive organizational context for women participants? Table 11. Gender Comparison of the Frequency of Accomplishments and Discoveries Cited Question 51: Are there any specific achievements or discoveries or contributions that you have made that are directly linked, in your view, to the knowl￾edge, skills, and new attitudes that you said you acquired in the United States? By Gender Yes % No % Total % # Male 78.8 21.2 100 978 SE 1.3 1.3 Female 69.4 30.6 100 543 SE 2.0 2.0 Both Sexes 75.3 24.7 100 1569 SE 1.1 1.1 (Non-responses: 352 [18.3%] out of 1921) SE=Standard of Error 2. The Language Difference Fifty-three percent of the total number of participants came from Anglophone countries, 41 per￾cent were Francophones and six percent spoke Portuguese. In a sense, language is short-hand for the cultural differences that are widely recognized to exist between the three large “colonial” language groups in Sub￾Saharan Africa. The educa￾tional systems in Africa have distinct cultural-linguistic characteristics, each one re￾flecting “learning” styles im￾ported to Africa from Eng￾land, France, or Portugal. Learning and teaching styles differ as do the relative em￾phases placed, for example, on writing skills, the theo￾retical versus the practical or support for independent re￾search. Table 12. Comparison of Extent of Application of KSAs at Work by Language Group Question 38: If you answered “yes” to Question 36 above, how much of the knowledge, skills, or new attitudes ... did you apply in your work? By Language A great deal % A lot % Some % A little % No Opinion % Total % # English 68.7 28.9 1.9 0.5 0.0 100 1149 SE 1.4 1.3 0.4 0.2 -- French 53.1 39.9 5.4 0.8 0.0 100 614 SE 2.0 2.0 0.9 0.4 0.4 Portuguese 51.9 31.9 16.3 0.0 0.0 100 135 SE 4.3 4.0 3.2 -- -- Total 62.4 32.7 4.1 0.6 0.3 100 1898 SE 1.1 1.1 0.5 0.2 0.1 (Non-responses: 23 (1.2%) out of 1921) SE=Standard of Error To what extent were there differences in impact from the perspective of these cul￾GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 32 Section I: Findings tural-linguistic groupings? If there were differences, what were the consequences? In what way might future programs consider language in allocating resources across Africa, or in designing effective follow-up to leverage impact? Quantitative data gathered from the survey offered a few insights. To begin with, there were only fractional differences according to language in Level 2, participant acquisition of skills. But when queried as to the amount of KSAs they could apply (Level 3), participants revealed slightly more differences, as shown in the table for survey Question 38. Portuguese speakers reported somewhat less success at applying their new skills than French or English speakers. When asked about the level of difficulty or ease they encountered in applying their KSAs, lan￾guage increased considerably in importance, as shown in the table for survey Question 41. Over 11 percent of the Francophones found their applications at the workplace “very difficult or im￾possible” compared to only one percent of Anglophones, while 63.7 percent of Portuguese speakers selected “possible but difficult.” The variations were large enough to lower the totals, which are weighted averages of the columns above, so that “possible but difficult” received the highest number Table 13. Comparison of the Degree of Difficulty in Applying KSAs at Work by Language Group Question 41: Please indicate how difficult or easy it was to apply your knowledge, skills, and new attitudes where you worked immediately after returning. By Language Very Easy % Fairly Easy % Possible but difficult % Very Difficult % Impossible % Total % # English 32.9 36.2 29.9 1.0 0.0 100 995 SE 1.5 1.5 1.5 0.3 -- French 42.4 21.5 26.4 6.4 3.3 100 516 SE 2.2 1.8 1.9 1.1 0.8 Portuguese 10.4 21.5 63.7 0.0 4.4 100 135 SE 2.6 3.5 4.1 -- 1.8 Total 34.0 30.4 31.6 2.6 1.4 100 1646 SE 1.2 1.1 1.1 0.4 0.3 (Non-responses: 275 (14.3%) out of 1921) SE=Standard of Error In terms of sharing knowledge with others (Q42), applying KSAs outside the workplace (Q45) or noticing institutional impact (Level 4, Q47), no differences emerged by language group. Even in terms of making any specific discoveries or contributions (Q51), language appeared to play no significant role. But when asked to indicate the levels where the changes they described took place (Q49), these differences appeared, in order of ranking in the following table. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 33 Section I: Findings Table 14. Comparison of the Location of Impact by Language Group Language 1 2 3 4 English Institutional National Sectoral Community French Institutional Community Sectoral National Portuguese Community Institution National Regional The quantitative data suggests that English-speaking Africans had the most ease in transferring their new knowledge, skills, or attitudes to their workplaces (for the combined “very” and “fairly” easy columns, compare 68 percent of Anglophones, 58 percent of Francophones, and 32 percent of Lusophones). Their success rates at applying these skills were higher, which is under￾standable if they had an easier time, as indicated. Overall, Francophones succeeded more often and more easily than Portuguese-speakers. The site visits to three countries using English, three using French, and one using Portuguese un￾covered no evident explanations or verification that one language group had more or less institu￾tional impact, for example, than another. Since one country represented Portuguese primarily in the data set (Mozambique), its particular historic and institutional setting may throw more light on its lower “rating” in terms of ease of application of KSAs, than the others. Mozambique en￾tered independence with a heavy dose of statist organizational culture due to the Marxist political ideology that dominated the war period. It is entirely possible that leadership in the country’s principal institutions was quite resistant to change threatening the status quo. Ideas brought back from the West, which had fought bitterly against Mozambique during the liberation war, might emphasize expressive freedoms, individual research, and intellectual dialectic over social ad￾vancement. One striking example of extraordinary impact, however, came from Mozambique, albeit nearly a decade after independence. Not only did the returned participant have to establish a new journal￾ism department, he had to justify that the university needed one! This case tends to support the data findings that returned Mozambiquans (all of whom were the ATLAS grantees from the 1990s) confronted less conducive institutions upon their return than those from other countries. It would appear, then, that this phenomenon was due more to the country’s special circumstances and less to the cultural-linguistic characteristics associated with a Lusophone country. Concerning the often-discussed Anglophone/Francophone cultural divide, anecdotal information culled from site visits turned up no particular explanations. Both sets of countries had institutions that greatly benefited from ATLAS/AFGRAD training. Madagascar’s Ministry of Education, whose minister is a returned ATLAS participant, has undergone tremendous modernization due to his leadership. Mali’s applied agricultural research has been significantly advanced by re￾turned participants. Likewise, Ghana’s flagship university at Legon now boasts retired AT￾LAS/AFGRAD professors who have spent their entire post-graduate careers in teaching and curriculum design. One element frequently cited by Francophones in responses to open-ended questions is the extent to which learning English improved their opportunity for impact. Participants regularly cited GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 34 Section I: Findings their ability to access international publications, surf the Internet, and attend conferences—in other words, strengthen linkages in their field—through the vehicle of English. Being able to ac￾cess currents in their field in English increased their impact at home and, in some instances, be￾yond. 3. Type of Institutions Programs designed to increase “human capacity” prior to 1990 targeted public-sector institutions, in particular universities and government ministries working in key development sectors (primar￾ily health, agriculture, and education). In tandem with the dismantling of government-directed economies and the emergence of a larger private sector, USAID shifted somewhat to include new types of organizations. The term “private” included for-profit companies (in particular, SMEs and micro-enterprises), nongovernmental organizations (NGOs) working in development sectors, and professional and business associations. In terms of extent of impact, were there differences due to the types of institution that employed U.S.-trained participants after training? If so, what might explain these variations? When tabulated by type of institution, an￾swers to Question 47 (...has there been any difference in your institution’s out￾put...?”) provide a perspective. The most present types of institutions (for-profit, NGO, and public sector) have report little difference in institutional output change as a result of KSAs being applied. What is surprising is the lower level of change re￾ported by participants working in donor agencies. The primary variable, it would appear, between the two categories is the degree of control exercised over the insti￾tution. In the first two (businesses and NGOs), participants are most likely either owners or decision-makers. In public sec￾tor institutions, despite their weaknesses, participants in many cases can and have made a difference, as shown elsewhere in this report. But in a donor agency, such as USAID, the United Nations affiliated or￾ganizations, the World Bank, or African Development Bank, participants may have less influence in making changes. Or, their transfer of the fruits of training to these institutions does not have as direct an effect on output as with the others. Table 15. Comparison of Differences in Output by Institution Type Question 47: If you were able to apply your knowledge, skills, or new attitudes at work, ... has there been any differ￾ence in your institutions’ output (in terms of quality, quan￾tity, and so forth), performance, productivity, or impact? In other words, did anything change? By Type Yes % No % Total % # For-profit business 100.0 0.0 100 423 SE 0.0 0.0 NGO 97.8 2.2 100 135 SE 0.9 0.9 Public 95.1 4.9 100 137 SE 0.8 0.8 Donor 81.2 18.8 100 69 SE 4.7 4.7 Self-employed 83.9 16.1 100 56 SE 4.9 4.9 Retired 100.0 0.0 100 32 SE 0.0 0.0 None—unemployed 71.4 28.6 100 7 SE 17.1 17.1 Other 90.5 9.5 100 116 SE 2.4 2.4 (Non-responses: 38 (21.5%) out of 1921) GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 35 Section I: Findings Although USAID never set out through ATLAS/AFGRAD (or any other human capacity devel￾opment program) to train USAID’s own present or future staff, U.S. graduates are nonetheless found in every office in every country. Similarly, the World Bank and U.N. field offices are well-stocked with U.S. graduates. The reasons for this employment phenomenon are known and do not bear directly on this study. That they perceive less change in these vital development in￾stitutions due to their training is a finding that bears further analysis and reflection. Anecdotal information gleaned from site visits to the very institutions that benefited from dec￾ades of graduate training for their key personnel reveal little distinction based on the categories of institutions shown above. However, African public institutions have suffered greatly over the years in declining resources, low morale, and departing cadres, to an extent that the high level of impact cited by participants in these institutions is noteworthy. On the other hand, the universi￾ties visited by the team, it must be said, were vibrant and growing: Ghana’s universities (Legon, Cape Coast, and so forth), Mozambique’s Eduardo Mondlane University, Uganda’s Makerere, and Mali’s research institutes have all welcomed back significant numbers of U.S. trained pro￾fessors. 4. Significance of Degree Level and Year of Completion The ATLAS/AFGRAD programs sponsored African professionals for graduate study that led to master’s (MPh, MBA, MS, and so forth) and doctoral degrees. Although designed to sponsor an applicant for only one degree, a number of participants continued to a higher degree (often funded from other sources) after the master’s or received PhD funding after completing an “un￾funded” MA. For the purposes of this assessment, were there differences in impact according to the degree obtained? Although many factors can be used to assess the value of a PhD in addition to impact, a decision to fund a participant for a PhD, at double or triple the cost of a MA pro￾gram, should ideally result from a careful analysis of the African institution’s specific require￾ment for a person with doctoral-level knowledge and skills. Merely assessing an individual’s ability or commitment to complete the degree is insufficient. Many factors weigh into an African MA degree graduate requesting continued funding to the PhD, such as “degree obsession” at home, individual worth (“I excel in my field and therefore deserve the PhD”), or the fear of not competing well with peers at the home institution. These do not address USAID’s objective to obtain measurable institutional or sectoral impact from its investments in human capacity devel￾opment. A corollary to assessing the relative impact of degree level is to consider differences according to the year in which the participant obtained the degree. Answering this question might shed light on the length of time needed for impact to take root or spread. a. Significance of Degree Levels Analyzing Question 47 by degree, there emerged no significant difference in institutional impact. Ninety-five percent of PhD holders affirmed that their institutions changed against 96 percent for GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 36 Section I: Findings those with a MA, 91 percent for an MBA, and 98 percent for a MPh. The weighted average for all degrees combined answering “yes” to Q47 was 95.7 percent, with PhD degrees representing 38 percent of the total, MAs at 39 percent, MBAs at 8 percent, MPh at 5 percent, and “other” at 6 percent. The weighted average of those responses stating there was no institutional impact was 4.3 percent. If the sectors or fields of study of the degree-holders were analyzed, would differences emerge? Since there was not an equal distribution of PhDs and master’s in each field, a simple impact comparison by degree is not possible (for instance, few PhDs but many MAs were in manage￾ment, thereby corrupting any comparison). Since the affirmations of impact were so high, and the distribution between degrees so unequal, the best analysis was to identify the field where the most degree-holders claimed not to have had impact. The only sector (other than “Other,” which had 10 each for both degree-holders) selected by PhDs where they claimed no institutional change (Q47) was Agriculture, where 18 doctoral graduates answered “no.” With the master’s graduates, none chose Agriculture but 8 (out of 877) chose Health and 7 chose Transportation. Perhaps the most salient point is that the only sector not to be singled out for no impact by either graduate group was Education. Since the number of respondents answering negatively was low, any observations made have to be cautious. Considering answers to Question 49 about the level at which their purported impact occurred (in￾stitution, community, sector, and so forth) would help clarify and verify answers to Question 47. The priority rankings shown in Table 15 below compare participants who obtained a PhD degree (regardless of funding source) with all those who obtained MA degrees. Table 16. Comparison of the Location of Impact by Degrees Obtained DEGREE 1 2 3 5 5 6 PhD Institutional Community National Sectoral Regional International MA Institutional Sectoral National Community Regional International Although doctoral degree-holders believed the community benefited from the impact they claimed to have had whereas the master’s degree-holders pointed to their sector, in reality few differences emerge from the above analysis. Furthermore, the statistical differences were slight between the various choices, diminishing the significance of the different rankings shown. b. Significance by Year of Degree Completion To analyze the possibility that the year of degree completion affected impact, participant answers to Question 2211 were divided by decade and cross-tabulated with Question 51 that queried about any achievements that participants made that were directly linked to their acquired KSAs. The 11 Q22: In what year did you receive the highest diploma indicated above? GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 37 Section I: Findings table shows quite clearly that impact increased over time, that participants obtaining their de￾grees in the 1970s had significantly higher af￾firmations of impact than those from subsequent years. To conclude, however, that “impact takes time” based on this data analysis would ignore the possibility of other factors. For example, did the fields of study shift from decade to decade (for instance, education to agriculture) or institu￾tion types (public-sector to private)? Did the countries in the participant selection pool change, leaving countries with institutions less conducive to capitalizing on the returned partici￾pants’ skills than before? Without the luxury of isolating these various exogenous variables from the analysis, the most that can be stated, based on the data, is that impact did increase over time. This observation substantiates what human re￾source development specialists frequently claim when confronted with measuring the return on capacity-building investments: impact takes time and the 5-year activity life cycle at USAID works against investments in long-term training. Table 17. Comparison of the Frequency of Accomplishments and Discoveries Cited by Degree Completion Year Question 51: Are there any specific achievements or discoveries or contributions that you have made that are directly linked, in your view, to the knowl￾edge, skill and new attitudes that you said you ac￾quired in the United States? By Decade Yes % No % Total % # 1970s 92.3 7.7 100 509 SE 1.2 1.2 1980s 68.3 31.7 100 460 SE 2.2 2.2 1990s 63.8 36.2 100 547 SE 2.1 2.1 Total 74.7 25.3 100 1516 SE 1.1 1.1 (Non-responses: 405 (21.1%) of 1921) 5. Individual Achievement and Career Advancement Individual achievement alone would not imply impact, as defined in this assessment (e.g., a par￾ticipant elected to public office by corrupting the political process). Likewise, rapid advancement within an organization due to economic class, nepotism, ethnicity or religion does not demon￾strate impact. But when respondents state they made specific achievements or discoveries, then provide viable examples that reasonably support their claim, it is likely that positive changes re￾sult from participant accomplishments. Three-fourths of the respondents stated that they made specific “achievements or discoveries or contributions” directly linked to the KSAs acquired in the U.S. A reading of the text examples given on the questionnaire (see Volume III, Annexes C and D) provides ample indication of the significance and quality of individual achievements. Some of these are recaptured below: • Developed new improved food legume varieties • Discovered the existence of threshold viscosity in consolidating saturated clays • Published articles on andragogy with a renowned American adult educator • Contributed to the development of improved cultivars in Africa • Researched and published A Guide to Cotton Cultivation in Ghana GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 38 Section I: Findings • Published many educational articles for the Calica Journal and the English Teaching Fo￾rum • As an employee of CNN in Africa, motivated African TV stations to send in stories rep￾resenting alternative viewpoints • Designed and started the first architecture curriculum at the university • Actively participated on a team that challenged the Divorce Act in a constitutional peti￾tion • Started a “network” dealing with law, ethics and HIV/AIDS • Instrumental in changing banking law to allow micro-finance institutions to mobilize sav￾ings A striking example of persistence in applying the KSAs acquired during training, in the face of a hostile environment, is illustrated in the following case study, where a participant from Mozam￾bique succeeded in establishing a department of communications at the national university. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 39 Section I: Findings Impact Example 7 How Independent Analysis Led to National Impact (but only after overcoming internal resistance) Eduardo Namburete Dean of the Faculty of Communication and Art. Maputo, Mozambique After Mr. Namburete took his master’s at Southern University in 1998 in communications, he did a one-year stint as a foreign reporter for the Washington Times in Washington, D.C. He had come to realize during the late 1990s, dur￾ing the transition from civil war to peace at home, the critical role played by the press in developing democratic practices. In the United States he began to question why a stronger independent core of journalists had not material￾ized in Mozambique. He concluded that the problem was the absence of a department of communications at the Mo￾zambique’s Eduardo Mondlane University. After returning home in 1999, he began a persistent effort to establish a viable communications department from his position as Director of Public Relations. Confronted by immediate resistance to the idea of a communications department from colleagues who placed greater priority on medicine, law, and more “development-related” fields, the participant systematically built his case. He first conducted an independent feasibility study to counter the skeptics who believed that no students would apply for such a degree. Others warned that government, so recently emerging from the statist traditions of Mozambique prior to the end of the civil war, would not fund a degree program whose graduates would criticize its policies. Oth￾ers encouraged him to push on. After a few years of presentations and debate, the university approved the degree program and the doors were opened. Over 1,000 applications were received in February 2003 for 50 places in the program, making not a few heads turn in amazement. Mozambique had established the country’s first independent communications degree pro￾gram due to the tireless efforts of a returned participant with a master’s degree in communications and a briefcase full of communications tools collected during a rich and varied experience in both academia and as a reporter. The contribution of the ATLAS Program The participant states the training received at Southern University and the work experience at the Washington Times gave him confidence and a mission that strengthened his resolve to succeed. Of course the graduate education and knowledge gained were important, but according to Mr. Namburete, the time spent in the United States, that even included covering the White House, bolstered his self-assurance. As he put it succinctly, his long-term training was the “watershed in my life.” At the same time, Mr. Namburete took great pains to credit his colleagues who believed deeply in this mission, for the success, a recognition of the power of teamwork over individual ambition. Note: The above account, recorded by Assessment Team Member Hap Carr, was subsequently cross-checked with one of the colleagues with whom the participant collaborated for accuracy. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 40 Section I: Findings Evidence of career advancement was obtained from the Internet Search where a high percentage of the randomly selected participants occupied positions of considerable responsibility, as dis￾cussed under the section above (“Regional and International” findings). Answers to survey Q54 (To what degree do you attribute your present income (or economic circumstances) to your par￾ticipation in the AFGRAD or ATLAS programs?) further substantiate the finding that participants advanced in their careers due to the training received. Participants who create, expand, or manage a private for-profit company could be said to have achieved something significant and, by extension, had an impact. Survey Question 55 looked into this aspect and found that 32.4 percent of the respondents answered affirmatively. Given that only a small number of participants worked for a for-profit employer prior to their U.S. degree program, the percentage of returned participants claiming to be active in the sector is noteworthy. Anecdotal information gathered from site visits turned up many participants who founded and ran businesses on the side, in addition to their professional roles in a targeted institution. How much income was generated by these businesses? How many employees benefited from this in￾come? What effect did the businesses have on development? Would these businesses have been created with the participant’s exposure to the United States? Nearly one-third of the estimated total of 1,921 participants represented in the survey affirmed to be actively involved in the for-profit sector, yet were not selected from that sector prior to train￾ing. This unanticipated by-product of a long-term U.S. academic training program merits serious analysis to inform future long-term programs. When participant achievement and career advancement are the main focus of an assessment, the results appear as anecdotal success stories and cannot be extrapolated to deduce impact. But when individual accomplishments and indications of participants holding positions of influence are confirmed through a variety of quantitative and qualitative techniques, including grounding of their answers, it is reasonable to conclude that the training received contributed significantly to important achievements that, in turn, induced impact at the institutional, sectoral, community, regional, and even international level. F. Identifying Areas of Special Impact 1. Participant Whereabouts The assessment concentrated on obtaining information primarily from a sample of participants from 21 countries, either queried by questionnaire or interviewed directly in-country, on per￾ceived impact. A tracer study was not conducted to identify what had become of the 3,219 par￾ticipants known to have graduated from U.S. universities under the two programs. Questions 31 through 34 attempt to learn about participant whereabouts as an indication of the potential to have an impact. Merely holding a position in an organization, as stated elsewhere in GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 41 Section I: Findings this assessment, does not imply impact. But were the majority of the participants to become un￾employed upon graduation, the likelihood for impact beyond the individual declines. Question 31 asked since your return from the U.S. after your ATLAS or AFGRAD experiences, have you worked for the same institution where you worked prior to leaving your country? Nearly half (49.4 percent) of the participants surveyed returned to work at the same institution where they were employed prior to going to the United States. This appears to be an impres￾sively high number, given the ups and downs of many African institutions over the years and the political instability that thwarted the return of many participants (to Ethiopia, Uganda, Ghana, Liberia) for years. The percentage includes recent returnees as well as retired alumni. Twenty￾five percent of the respondents stated that they worked for a different organization, with 17 per￾cent indicating the question was not applicable or that they were not working. Of those who indicated they did not work for the same institution, the results showed: • 21 percent worked for one organization (but not the same one prior to training); • 42 percent worked for two organizations; • 28 percent worked for three organizations; • 5 percent worked for four organizations; • 2 percent worked for five organizations; and • 2 percent worked for six organizations. Although these averages are deceptive since they include recent returnees as well as retired par￾ticipants, the data clearly supports a view that the ATLAS/AFGRAD participants remained in relatively few institutions. A normal career path would show at least two or three institutional changes, placing the participants well within the range of what would be considered a stable work environment. Since institutional change requires a minimum of work stability, it appears as if that prerequisite for change was easily met. On the other hand, given the dearth of institutional choice and employment opportunities in all African countries, the lack of mobility is not surpris￾ing. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 42 Section I: Findings Table 18 Examples of Current or Past Positions Held at the World Bank Group (all are former AFGRAD participants) Year of Name Position Degree Mr. Koffi Edoh Senior Economist Consultant, WB 1983 Mr. Mamadou Dia Country Director (various), WB 1969 Mr. Joseph Jones Alternate Executive Director, WB 1976 Dr Ruth Kagia Director of Education, WB Mr. Joseph Kakoza Deputy Division Chief, IMF 1973 Mr. Doe Lubin Senior Economist, IMF 1979 Mr. Callisto Madavo Vice President, Africa Department, WB Mr. Nlandu Mamingi Development Economics Consult., WB 1992 Mr. Gobindram Nankani Vice President, WB 1976 Mr. Rudolph Polson Senior Economist, WB 1989 Mr. William Saint Senior Education Specialist, WB Mr. Pierre Sooh Civil Engineer, WB 1968 As previously cited, of the 51 ATLAS/AFGRAD alumni discovered in the Internet search, 34 former participants were found to be working in the United States or Canada (16), in other Afri￾can countries (8), and in international organizations around the world (10). 2. Participant Contributions outside the Workplace Ninety-four percent of the respondents indicated that they applied their knowledge, skills, and attitudes acquired in the United States outside the workplace. There were no differences when disaggregated by gender. When language is considered, fewer French-speakers (88.1 percent) reported such contributions compared to 96.9 percent and 94.8 percent for English and Portu￾guese speakers, respectively. But the significant finding is that such a high percentage of respondents affirmed that they used their KSAs in non-work situations. Some of the examples cited in survey Question 46 by the participants are: • Helping others to set up their private businesses • Joined an association through my church that deals with mental problems • Worked on multidisciplinary teams to improve neighborhood life in Nampula • Helped my wife run a small business • Used the participatory approach to solve community problems • Served on community boards in the health and human rights sectors • Mentored young people the way I witnessed it in the United States GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 43 Section I: Findings • Helped graduates establish themselves as entrepreneurs • Share family duties with my wife • Participated in teaching about the dangers of HIV/AIDS to communities 3. HIV/AIDS Work Although this assessment did not focus on any particular aspect of HIV/AIDS, it uncovered many examples of participants working in or around this area. A significant percentage of the graduates in health can be presumed to have worked or continue to work in areas that impact on HIV/AIDS. Several examples emerged showing participant impact in fighting HIV/AIDS that best demonstrate how participants are applying their KSAs to the pandemic. “As Founding member of the Uganda Network on Law, Ethics and HIV/AIDS, I organized the training of 23 Districts in the legal dimensions of HIV/AIDS for their communities. My goal was to have legislation and networks in 23 dis￾tricts dealing with these issues. I helped publish research that led to the adop￾tion by the Law Reform Commission of changes that were in turn incorpo￾rated into Uganda law that improved the legal environment for HIV/AIDS.” Regina Mutyaba, Attorney, Uganda As a graduate in medical technology, Ms. Marcelle Mireille Dehoue had not thought much about applying her knowledge to the fight against HIV/AIDS. She had returned to Benin and after a few years of struggle to get an appro￾priate position, was asked to organize the rehabilitation of the food microbi￾ology lab (which tests the quality of the shrimp that ECOWAS exports to the European Union). One of her passions, however, was the role played by tradi￾tional healers in the HIV/AIDS pandemic. Her enthusiasm in applying her medical technology background to HIV/AIDS led her to write a research pro￾tocol on HIV/AIDS that addresses the biological difference between ap￾proaches used by traditional healers compared to those associated with mod￾ern medicine. Marcelle Mireille Dehoue, Benin The assessment was not able to disaggregate participants working on HIV/AIDS issues, either directly as professionals in the field or indirectly out of personal commitment. However, team members met many participants who mentioned their work with HIV/AIDS in individual impact statements, on questionnaires, and in interviews. It was not possible to determine what impact the ATLAS/AFGRAD programs had on HIV/AIDS, or to decipher the effect of HIV/AIDS on the participants, given the resource and time limits of the assessment. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 44 Section II: Scope of the Assessment Assessments have periodically looked at the impact from investments in human capacity devel￾opment, typically concentrating on the contributions made by returned participants to their coun￾try’s advancement. In the early years, human resource specialists considered the following fac￾tors in concluding whether the training attained its intended objectives: • Have the participants returned to the same employer and subsequently assumed positions of greater responsibility? • Has the project trained the number of individuals targeted, were qualified beneficiaries selected, and did they complete the training “successfully?” • Are there sufficient numbers of U.S.-trained professionals locally with whom U.S. offi￾cials can find common ground in promoting development, resolving international prob￾lems, and maintaining healthy dialog? To measure the extent these criteria were met, evaluators would conduct linear “tracer” studies to track the career path of returnees. They would administer surveys to determine the participants’ views of the training and highlight (with some degree of pride) the returnees who occupied im￾portant positions in government and civil society. When participants were unable to return, usu￾ally due to instability at home (for example, at various periods since the 1960s, in Eritrea, Ethio￾pia, Ghana, Guinea, Liberia, Mauritania, Somalia, and Uganda), they were often found in inter￾national or regional organizations. The prevailing assumptions were that investments in long-term U.S. training were “worth it” and that such programs solidified U.S. “friends” in economically emerging countries. In the 1991 A Training Impact Evaluation Methodology and Initial Operational Guide, the Education and Hu￾man Resources office of the Africa Bureau raised fundamental questions about the value of pre￾vious assessments: Without a theory, human resources development will continue to be, as it has for decades, an act of faith reflecting the maxim that education is an in￾trinsic good.... Without a theory, donors and host countries will continue to tinker at the margins of the existing system. (p. II-15) The guide laid the groundwork for subsequent development through the 1990s of impact assess￾ment methodology. The questions driving USAID and its partners were: What is the best way to determine “impact” from sizable investments in human capacity-building? What acceptable ap￾GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 45 Section II: Scope of the Assessment proaches have emerged to finding out whether it was “worth it” (beyond the “intrinsic good” of improving an individual) to train thousands of African professionals? How can evaluators go be￾yond recounting success stories to tap more quantitative data to assess impact? A. Objectives and Definitions The objective of this study is to assess whether any impact occurred from long-term U.S. aca￾demic training provided to participants in the ATLAS/AFGRAD projects. As cited in the Execu￾tive Summary of this Report, for the purposes of this assessment, impact is defined as: Any change that occurred at the institutional, sectoral, national, or regional level attributed to ATLAS/AFGRAD-sponsored training. The ATLAS project overall purpose, as stated, was “to strengthen leadership and technical abili￾ties and enhance professional performance of individuals serving in African public and private sector entities, including universities, research centers, and other key development institutions.” The four goals cited in the ATLAS Scope of Work most relevant to this assessment were:12 • strengthened programs in educational and training institutions, particularly in scientific, technical, and economic fields; • research institutions expand and improve their human capacities to carry out research relevant to African development, particularly for increasing agricultural productivity and technologies; • public sector institutions show improved equity and efficiency in providing key services (health, education, transportation, and so forth); and • increased capacity among women to fill leadership and non-traditional roles. Some of the indicators listed to measure achievement of these goals included: • employment of the individual in key African development-related institutions or in pro￾ductive private enterprise; • level of authority and responsibility and promotion record of the individual; • important personal accomplishments on the job; • immediate impacts of the individual’s actions on organizational decisions; • authority and influence of the individual as perceived by knowledgeable others; and • the performance of female graduates compared to that of males. 12 The new SOW in the Annexes includes the complete list of goals and indicators from 1990. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 46 Section II: Scope of the Assessment The goals and indicators above drove the design of an “impact assessment framework,” intro￾duced in the next section, that would help collect and analyze the information gathered. With the above original goals and indicators in mind, the team designed an “impact assessment framework” to zero in on those indicators that could reveal characteristics about change. For ex￾ample, “employment” (from the list above) was questioned as a useful indicator since counting the number of participants occupying civil service positions, for example, yields little informa￾tion about impact. On the other hand, the “level of authority” indicator could imply at least the possibility that the participant effected change. The methodology included ways to ensure that female participants were selected, interviewed, and included in the data gathering to be able to compare male with female “performance,” which the team interpreted to mean “impact.” The resulting participant questionnaire gathers data respecting these indicators and framework. The framework employed for this assessment is a modified version of the widely used Four Lev￾els of Evaluation (Donald Kirkpatrick), which is described in detail in Volume II, Annex C. The analysis of the data also followed this framework. The ATLAS project targeted African institutions, rather than qualified individuals, for capacity￾building. One of the hallmarks of the AFGRAD project was its commitment not to continue the brain-drain that had been criticized in the predecessor ASPAU program, which was terminated in the 1970s after evaluators disclosed a 35 percent “repatriation” rate.13 The focus on institutions helped the team rely on an impact assessment framework. The assessment SOW tasked the team to visit four U.S. universities that participated in the pro￾gram. In order to focus more on obtaining evidence of impact from the field, the team recom￾mended that this task in the SOW be dropped. This decision freed up resources that were used, for example, to add a seventh country (Francophone) to the site visits. (For a complete list of modifications, see Scope of Work and Modifications appended to the new SOW in Volume II, Annex J.) B. Data Collection Methodology The modified Kirkpatrick framework served as a guide in designing the survey questionnaire so that questions could be grouped by Impact Level. This provided a structure to the discussion of impact in Section II. Four collection methods were employed that could be undertaken within the time and resources the team had available. 13 African Scholarship Program for American Universities, one of the early USAID-funded programs that AAI ad￾ministered in the 1960s for undergraduate African students at the U.S. colleges and universities. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 47 Section II: Scope of the Assessment • Participant Survey—a statistical survey of a weighted randomized sampling of partici￾pants that included both quantitative and qualitative data. • Country Site Visits—that included participant group meetings, individual interviews and selected institutional visits in 7 countries. • Internet Search for impact from participants in countries not visited by the team. • Interviews with AAI officers, USAID/Washington education specialists, USAID Mission staff, African institutional leaders, and supervisors of participants. The methodologies are summarized in the following discussion; for more details on each of these methods please refer to Volume II, Annex D. 1. Participant Survey To uncover as much evidence as possible within the resources and time available, the team cre￾ated a representative, random sample drawn from the data base provided by AAI of ATLAS/AFGRAD participants from 45 Sub-Saharan countries. It was decided early on to cast the net across the continent rather than select certain countries where a priori face-to-face inter￾views could be conducted. The team saw an opportunity to take impact assessments one step fur￾ther by designing a participant survey with a degree of statistical integrity that would allow for a more grounded analysis of impact that could lead to data-justified findings. The steps used to build the sample began with a thorough review of the AAI database that con￾tained records on all the participants from the 45 countries through the formulation of randomly selected country and participant lists. After 27 countries were randomly selected, 671 partici￾pants were randomly selected from these 27 countries as “core” participants to locate for the sur￾vey. After administering the survey, to accommodate changes encountered, the “original survey universe” was modified to 21 countries, as shown under “revised survey universe” in the table in the Annexes (Volume II, Section D). The weights were then applied to the sample as well as the relative standard error calculations. A survey questionnaire (see Volume II, Annex E) was developed to administer to the participants chosen, following the steps above, by email, fax and during the country site visit meetings. Dur￾ing the process of building the sample and administering the questionnaire, the team arrived at a universe containing certain characteristics.14 The 203 completed questionnaires from 21 coun￾tries were statistically weighted to represent 1,921 estimated participants with relative standards of error. The overall weights of the “revised sample” (that is, the result obtained by “weighting” the 203 completed and returned questionnaires) compares well to the “original universe” (that is, the actual proportions of the 3,219 participants) filtered by the three variables—language, gen￾der, and program. For instance, 58.6 percent of the revised sample is English-speaking compared 14 A more technical description of the methods used to produce the sample is included in the Annexes, Volume II, along with a table entitled Characteristics of Universe and Sample. Each country is listed in tables with their respec￾tive weights, the number of participants selected and the number of participants in the “revised universe and the “country weight” for each selected country. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 48 Section II: Scope of the Assessment to 53 percent of the actual universe. Most of the other categories showed a similar proximity be￾tween the “original universe” and “revised universe.” Women were slightly overrepresented in the data used in this assessment (22.4 percent in the survey compared to 19.9 percent in the uni￾verse). The relative standard of errors, discussed under “Reliability of the Findings” (Section I.A of this Volume), are low in general. 2. Country Site Visits The main purpose for a site visit was to collect information about impact in a different way than accomplished through the questionnaire-based participant survey. Arranging face-to-face struc￾tured meetings with participants in a group setting would enrich the assessment by eliciting anecdotal information about impact. Not only would individual examples emerge, but more importantly, the team could follow leads about the characteristics of the impact that the quan￾titative survey might overlook. A careful analysis was conducted that led to the selection of seven “site-visit” countries so that a balance was en￾sured in terms of program (ATLAS or AFGRAD), num￾ber of participants (only countries with large numbers of participants), language, and region. The methodology fol￾lowed in selecting these seven countries is presented in detail in Volume II, Annex D. The team divided into groups and conducted the site visits, each of which lasted 4 to 5 days organized by a local coordinator. Table 19. Participants Attending Impact Study Meetings by Country Benin 19 Ghana 28 Madagascar 10 Mali 16 Mozambique 24 Namibia 25 Uganda 14 Total 136 Visits were conducted to institutions that sent staff for long-term training in order to gain further information about changes that the participants were affirming took place. Although it was not possible to conduct formal organizational assessments of each institution, team members were able to hear from different sources whether impact that occurred was linked to the return of U.S.- trained staff. It was also hoped that interviews with non-participants could also be arranged who were familiar with a participant’s performance before and after training to report on any changed behavior. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 49 Section II: Scope of the Assessment ent team that could be used to triangulate and “field test” data culled from the questionnaires. 3. The Internet Search: Surfing for Impact amount of impact, r even the level, but of unusual, significant impact that might go undetected. s from 37 countries on the Master List, the search recorded atches on 69 names, of which: • nfirmed “hits” that fulfilled the criteria for “impact” and were verified as par- Table 20 Balance Achieved in Site-Visit Country Selection AFGRAD ATLAS Region Language % % Benin 35 65 WA Fr Ghana 100 0 WA En Mali 70 30 WA Fr Madagascar 50 50 SA Fr Mozambique 75 25 SA Po Namibia 0 100 SA En Uganda 80 20 EA En Total Averaged 58 42 WA=West Africa; SA=South Africa; EA=East Africa; Fr=French, En=English, Po=Portuguese A by-product of the “focus group” was thought to be an opportunity to augment the number of completed questionnaires. For these reasons, the country visits provided essential, first-hand in￾formation to the assessm The assessment team decided early on to try to use the Internet as a mechanism to determine whether participants were having impact beyond themselves. The goal of the search was to dis￾cover whether the Internet could be used to identify significant contributions from participants that might escape discovery by other techniques. Since the team could visit only seven countries and hoped to obtain questionnaires from about half of the remaining twenty of the 27 selected, the Internet innovation might fill a gap. The team was concerned not about the o A seasoned researcher familiar with Africa and fluent in French and English developed a search strategy and approach to guide his way through the Internet to places where, perhaps, participant contributions could be identified or assessed. A random a list of 200 names that were not on the survey sample list was run, 100 of which comprised the Master List to find on the Internet.15 Of the 100 randomly selected name m 51 were co ticipants, 15 See a complete description of the methodology used in the Annexes GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 50 Section II: Scope of the Assessment • 18 could not be verified due to name confusion or insufficient information, and • no matches were found whatsoever for 31 participants of the 100. l in that it roduced plausible evidence of impact that would have not been captured otherwise. ences can be based regarding the value of investments in long-term U.S. training for Af￾ricans. 4. Interviews of its newsletters, Alumni Directory, and write-ups on alumni occu￾ying prominent positions. n countries where questionnaires would be collected from partici￾ants or site visits conducted. Findings from the Internet search feed into analyses at various places, such as the discussion about regional and international impact in Section I.D and anecdotal reports of ATLAS and AFGRAD alumni occupying prominent positions. Using the Internet was successfu p The Internet search proved a rich exercise that augers for more consideration, resources, and ef￾fort in supplementing other more traditional data-collection tools designed to measure impact. Alone it is insufficient, as is any single information-gathering instrument. But used in combina￾tion with other methodologies, the Internet can produce valuable information upon which plausi￾ble infer Although it was not in the scope of this report to assess the management of the program, the team nonetheless met with the president and chief financial officer of AAI to establish guidelines for cooperation in terms of data sharing and obtaining anecdotal information on alumni. AAI shared with the team copies p The team held three meetings with officials at USAID to elicit their input on the methodology being employed and for assistance in ensuring support from the field. Suggestions from these meetings were integrated into the implementation of this report in large measure. Support from the field was most noteworthy for countries the team visited. Both the team and the CTO sent emails to all USAID missions i p GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 51 Section III: Lessons Learned and Confirmed about U.S. Long-Term Training Many useful lessons can be drawn from four generations of long-term U.S. university training for African professionals who, as shown in the preceding chapters, made valuable contributions that spurred development in Africa. Below are a few that policymakers and program designers may want to consider in the event the U.S. government reinstitutes future variations of the AT￾LAS/AFGRAD programs. Lesson 1: Target institutions for capacity-building in key sectors. A major reason for the success of the ATLAS/AFGRAD programs was their commitment to in￾stitutional change over individual improvement. Not only does this approach conform to best practices for performance improvement, which looks for changes in the workplace that can in￾crease output and produce results, but it also leads to higher impact. Participants are more ma￾ture, have secure employment, and are more likely to return. Lesson 2: Invest in long-term U.S. graduate training to ensure development impact. To ensure sustainable impact, USAID must invest in long-term training in the United States. Short-term training is important in transferring technical knowledge and skills, but only through long-term training can developing-country professionals absorb the research skills, modified work attitudes, and improved critical thinking that are prerequisites for making a measurable dif￾ference in their home countries. Lesson 3: Pay attention to non-technical aspects of long-term training. That participants repeated and forcefully stated that work attitudes, critical thinking, and other “non-technical” tools (such as self-confidence) were major attributes of their training loudly calls for far more attention to these “soft” aspects of U.S. academic training than previously included in program designs. That these benefits accrued to the extent they did with little programming attention or USAID investment only underscores how vital they are. With appropriate program support, every future long-term participant will return with a toolkit of non-technical, manage￾rial, and attitudinal solutions to the myriad challenges to be faced at the workplace at home. If GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 52 Section III: Lessons Learned and Confirmed about U.S. Long-Term Training participants succeeded so impressively without a concerted effort from sponsors, future impact with such support could be exponentially greater. Lesson 4: Consider “critical mass” approaches where possible. Although the assessment could not be conclusive on the role played by having a cadre of U.S. trained staff at an institution, anecdotal evidence and some quantitative data support the notion that critical mass made a difference. Changes backed by a group of participants were more pos￾sible, more sustainable, and more effective. Due to costs, managers have to select participants strategically from institutions—and should recognize that the number of participants from an in￾stitution sent to the United States is less critical to impact than their area expertise, potential for inducing future change, leadership qualities, and comm itment. Lesson 5: Apply impa ct value-bas ed criteria to all plans to sponsor doctoral degrees. The assessment found that master’s and doctoral degree participants reported nearly the same levels of institutional impact. This raised important questions about the value obtained for USAID’s investment in the two degree programs. How is this value calculated? What should be the rigorous justifications for U.S. government funding of a doctoral candidate, at several times the cost of a master’s degree? How should the value of that investment be compared to that of the master’s degree candidate? How can program managers avoid being influenced by a partici￾pant’s qualifications for the higher degree, that the person merits the sponsorship due to intellec￾tual acumen or high academic achievement? What are the preconditions necessary to accept funding for a doctoral degree? Lesson 6: Assess the relative merits of in-country versus long-term U.S. training fairly. Although this assessment did not compare the value of different training options used by USAID, the reasons for the demise of long-term academic training cannot be ignored. The high cost of U.S. training is often mentioned first. Yet this assessment provides ample evidence that the cost of U.S. training measured by the impact (or value) obtained could be extraordinarily low. The tendency has been for USAID to consider the cost of training against the numbers of recipients, essentially viewing training as a benefit to be distributed to the largest number of local nationals. Instead of comparing the cost (for instance, $75,000) needed for a single academic program in the United States with the cost of training 50 junior accountants in a program in their country to computerize financial management, program planners should estimate the value of the anticipated impact to be generated by the trained participants, once they apply their new KSAs. Using the example above, were program planners to consider the cost of obtaining the impact rather than the cost of the training, the return on having thousands of women in villages trained year after year about HIV/AIDS or about their marriage rights because one participant acquired KSAs, applied them, and made a difference, far exceeds the return from the junior accountant program. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 53 Section III: Lessons Learned and Confirmed about U.S. Long-Term Training USAID program managers need new tools to be able to fairly and objectively evaluate whether to spend funds on in-country, U.S. short-term, or U.S. long-term academic training, using a ra￾tional estimate of the value of the changes they wish to induce by using U.S. long-term training as the mechanism. Lesson 7: Maintain contact with returned participants. The assessment discovered that women and participants from some sectors encountered greater difficulties in introducing changes in their workplaces than participants in other sectors. Follow￾up support should be factored into future long-term training programs that can leverage USAID’s investment to induce change. Such support cannot be considered short-term. Sustained, in￾country activities will need to be managed so that returned participants obtain help. The ATLAS program included some activities of this nature, but the assessment did not discern any notice￾able effect on impact from this component. Lesson 8: Select sectors carefully for impact-oriented long-term training. By focusing on placing a value on the desired impact, USAID planners begin the process of choices. Since critical mass can provide internal synergies and support for accelerating changes in targeted sector-based institutions, planners should consider limiting sectors for long-term trai￾ning. They can also use another technique to increase impact and decrease the non-return possibility: send a small number of participants from the same target institution to the same uni￾versity in the United States for graduate work. Concentrating on sectors and institutions, as AT￾LAS/AFGRAD effectively did throughout the years, can help focus and sustain changes. GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 54 Section IV: Participant Feedback and Professional Enhancement Training A. Ideas from the Participants Aside from the anticipated recommendations from participants that long-term U.S. graduate training be restarted, participants had the following comments, which they freely wrote on the back page of their questionnaires.16 • Create conditions to put people together to discuss specific issues and ways they changed after the training. • The impact of the program would have been higher if the reintegration of the graduates back home had been part of the program itself. • The career path of the graduate should be outlined beforehand. • The major impact of the program is the capacity to change the attitudes and goals [of strong systems encountered at home]. This will only be achieved by living and studying in the United States for at least one year. • The time needed to reside in the United States to obtain the degree is too long. There should be African regional programs organized by USAID and conducted by U.S. profes￾sors. • The program is great. However, it would be good to have a follow-up program to help re￾turned participants start their own business, for example. • Training needs to be reinforced by more networking and more interaction with U.S. offi￾cials. • The U.S. officials need to discuss with government to make sure that participants are em￾ployed in the public sector in positions where they can use their new skills. • I was highly disappointed to learn that the ATLAS program was discontinued. It was contributing a great deal to economic, social, and skills development of Namibia. This is important if we are to sustain democracy. • This study came along right when defining decisions have to be made with regard to the position of U.S.-funded programs in Africa. I hope the U.S. government will continue touching the lives of thousands of Africans, through their generous sponsorships. 16 Participants were asked on their questionnaires, “Do you have any specific recommendations or general com￾ments you would like to make in relation to this Impact Study?” GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 55 Section IV: Participant Feedback and Professional Enhancement Training GENERATIONS OF QUIET PROGRESS An Impact Assessment of ATLAS/AFGRAD Aguirre International, September 2004 Page 56 Many participants emphasized the need for more follow-up to support their efforts to apply their skills to change their institutions. Many pleaded with USAID to continue the program. B. Professional Enhancement Activities The professional enhancement components of ATLAS have been the subject of considerable at￾tention over the years due to their higher profile than the participant placement activities neces￾sary to manage a long-term academic program for Africans in the United States. As a percentage of the total funding, “professional enhancement” was small: Approximately $5 million was spent under ATLAS over a 10-year period. Assessing the impact of this particular component was not a major factor in the assessment, which looked more to identifying impact attributable to long￾term training, not short-term regional seminars or other follow-up services provided the partici￾pants (alumni directories, in particular). Nonetheless, several questions were included in the questionnaire. Seventy-eight percent of the respondents affirmed to have received at least one Alumni Directory. They selected the follow￾ing responses when asked what use they made of the directory, in priority order: • to contact friends; • did not use it; and • contacted specialists in my country in my area of interest. A number of regional seminars were organized under ATLAS for returned participants from both programs. Very few of the participants queried in the survey or at participant meetings attended one of the seminars, which typically would invite between 25 and 50 participants. During the course of exploring for impact by gathering participant information from different sources, the assessment team did not encounter any significant indication that the professional enhancement components enhanced impact. However, the assessment is not charged with deter￾mining the efficacy of various aspects of the management of a long-term participant training program. The assessment cannot pronounce on whether future U.S.-based training programs should include a component designed to “enhance” professional impact through support activi￾ties such as regional seminars, grants to local organizations, newsletters and directories, or “Dis￾tinguished Alumni Awards.” Generations of Quiet Progress The Development Impact of U.S. Long-term University Training on Africa from 1963 to 2003 An evidence-based impact assessment of the value obtained from major investments in graduate education for 3,219 African professionals by USAID and its partners in the ATLAS and AFGRAD programs Volume II Supplementary Background Information Prepared for: U.S. Agency for International Development Bureau for Economic Growth and Trade Education Office Washington, D.C. Prepared by: Andrew Gilboy, Team Leader Harry Carr Thierno Kane Robert Torene Consultants to Aguirre International Guided and Assisted by: Cristin Springet, USAID/EGAT/ED Prepared for the U.S. Agency for International Development by Aguirre International under the Global Evaluation and Monitoring IQC, Contract FAO-I-00-99-00010-00, Task Order 13 SEPTEMBER 2004 Volume II Supplementary Background Information A. Description of ATLAS and AFGRAD Programs B. Objectives and Definition C. Modified Kirkpatrick's Four Evaluation Levels D. Data Collection Methodology and Country Selection E. Participant Questionnaire (abbreviated version) F. Qualifications of the Data G. Internet Impact Search: Strategy, Findings and Recommendations H. Memo of January 11, 2004: Proposed Methodology I. Examples of distinguished ATLAS/AGRAD Alumni J. Scope of Work and Modifications ANNEX A DESCRIPTION OF ATLAS AND AFGRAD PROGRAMS ______________________________________________________________________________ Established at the moment of independence for many African nations, the USAID-funded AFGRAD Program (African Graduate Fellowship Program [1963-1990]), and its successor ATLAS Program (Advanced Training for Leadership and Skills1 [1991-2003]), came to a close last April, having traversed many well-known development challenges and obstacles. Through these four decades, the ATLAS/AFGRAD regional program, managed by the Africa-America Institute (AAI), trained over three thousand African professionals for PhD and MA degrees at U.S. universities in fields critical to their country’s growth. The goal of AFGRAD program in the early years was oriented toward assisting young African nations with a supply of trained mid- and upper-level “manpower” in key sectors needed for development. In many cases, U.S. graduates replaced expatriates in key public-sector institutions. Subsequent iterations of the goal emphasized assisting African institutions build capacity. ATLAS continued this trend, introducing concepts of improving institutional performance and broadening the target institutions to include nongovernmental organizations (private companies, NGOs, and so forth). ATLAS also added the notion of leadership development to its purpose, and its name. The combined “waves” of USAID-sponsored graduate students administered through the ATLAS and AFGRAD programs ebbed and flowed for over 40 years. Through the three phases of AFGRAD and two of ATLAS, the Africa-America Institute managed the selection, placement, orientation, monitoring and follow-on for 3,263 Africans from 52 countries.2 Over the years, these regional programs were occasionally evaluated employing a methodology that focused on inputs (such as the number of participants receiving advanced degrees) or the whereabouts of graduates. The last evaluation completed in 1995, which was designed as a follow-up to a 1983 study, took a step further and reviewed participant impact based on a participant survey and small focus groups organized in four countries. This is the first assessment of the programs since the 1995 study. The total USAID funding for both AFGRAD and ATLAS was $182,585,026.3 Funding for AFGRAD I, II, and III was $107,201,231 and for ATLAS I and II $75,383.795. The U.S. Government portion of the total cost of both programs is estimated to be 85 percent, with the 1 Formerly the African Training for Leadership and Advanced Skills project. 2 The total participant entries in the AAI database for ATLAS and AFGRAD were 3,263 from 52 countries. The assessment team excluded from consideration 20 participants who did not complete graduate degrees, and 24 participants from outside Sub-Saharan Africa (Tunisia, Morocco, and several Caribbean nations), for whom awards were made for various reasons, to arrive at 3,219 participants for the survey. 3 The AFGRAD I and II accounted for roughly half of the three phases. ATLAS I was $27,455.214 and ATLAS II was $47,298,581. (The actual amount for ATLAS II will be slightly higher when closed out in 2004.) See the last page of Annex A for a conversion table into today's dollar values. remaining 15 percent paid by U.S. colleges and universities that waived tuition and African institutions that continued salary payments to participants while in training. The ATLAS/AFGRAD programs were administered more or less alike in terms of participant support, except that ATLAS included a larger "professional enhancement" component designed to increase impact by improving networks and linkages among returned participants. That component had the following activities: • Publishing and distributing regular "Alumni Directories" to enhance networking; • Publishing and distributing periodic newsletters to participants; • Establishing a Distinguished Alumni Award program; • Subscribing participants to professional journals after they return; • Organizing regional meetings in Africa on a variety of subjects to which participants were invited; and • Providing grants to African professional organizations of up to $30,000 "to nurture professional networks and direct critical attention to African's development problems." These follow-up activities were considerably more extensive and detailed in ATLAS than in AFGRAD, although some had been started under AFGRAD III in the 1980s. Both programs continued to award a small number of "post-graduate" grants to Africans to return to the United States to pursue a short-term, non-degree program to strengthen their expertise. Although this assessment is not concerned with the management of participant training, as it is called in the USAID lexicon, the ATLAS/AFGRAD programs depended on multiple funding sources over the years. Tuition "waivers" obtained by the contractor for each student reduced the level of USAID funding. African institutions, including governments often continued the salary of sponsored participants during their graduate programs in the United States. And in some cases, African governments provided the cost of air travel to and from the United States. These alternative funding sources for ATLAS/AFGRAD reduced the direct U.S. Government participation to approximately 85 percent of the total funding needed. This report recognizes throughout the important role played by USAID's "partners" in the programs' implementation. The two long-term programs were evaluated periodically throughout the forty years of operation. The abstracts of these evaluations are still available on the Internet and confirm that earlier studies focused on tracing the returned participants through their careers, calculating return ("repatriation") rates and assuming that impact was linked to promotions, salary increases and anecdotal reports. Little measurement of impact at the institutional or higher level took place, which was not unusual at the time. The most comprehensive retrospective of these large investments was completed in 19954 and highlighted AFGRAD's achievements. The MSI report found that "alumni are making a difference in key public and private institutions – introducing new techniques and skills, training others, introducing changes and innovations in their jobs, and contributing to the growth and 4 Capturing the results of 30 years of AFGRAD: Evidence of change in individuals and institutions across Africa – final evaluation report, Management Systems International, Inc. (MSI), December 1995. development of their organizations and institutions." It documented a high "return" rate of over 80 percent, with 90 percent completing their degrees. It stated that the participants' "upward professional mobility is a major contributing factor to capacity building in their home countries." Finally, the report singled out education as the principal sector where alumni had introduced the most lasting and far-reaching changes. The current assessment takes impact assessment methodology one step further by building a more reliable statistical foundation upon which to derive findings, as shown in Volume I, and covers both the ATLAS and AFGRAD programs. ATLAS/AFGRAD Program Costs in Today's Dollars Program Funding ($) CPI Multiplier 2003 ($) ATLAS (1995) 75,383,795 152.4 1.207 90,888,240 AFGRAD III (1985) 53,600,615 107.6 1.710 91,657,051 AFGRAD I & II (1975) 53,600,615 53.8 3.420 183,314,100 TOTAL 182,585,026 365,959,391 Note: This table shows the calculation used to arrive at the amount that would be required in 2004 to fund the same program, adjusted for inflation, and assuming tuition waivers from U.S. universities. The Consumer Price Index was determined to be the fairest method to arrive at the adjusted amount since most of the program costs were monthly living allowances, travel and administrative overhead—all well represented in the CPI. A single year was selected mid-way between each program's implementation to use as a base year. Another calculation used was based solely on tuition increases over the years that resulted in a $713 million price tag in today's dollars. The two calculations can be used to estimate what a new long-term training program would cost today, with and without tuition. On a per-year basis, the $4.5 million per month program cost cited ($182 million / 40 years) would be doubled to $9 million to replicate a new program with tuition waivers, and considerably more with no waivers. ANNEX B OBJECTIVES AND DEFINITION ______________________________________________________________________________ As stated in the body of the report, the objective of this study is to assess whether any impact occurred from long-term U.S. academic training provided to 3,219 African participants. Since the term impact is open to various interpretations, the assessment team decided on a definition to apply throughout the study: Any change that occurred at the institutional, sectoral, national or regional level attributed to ATLAS/AFGRAD-sponsored training. It was agreed that "change" could include deterioration (negative change) as well as improvement, although the assessment would not want to focus on negative impact. An objective assessment should not exclude the possibility of uncovering negative, or no, impact if driven by the data. In fact, during group meetings in each country, participants asked whether they should address any negative impact uncovered during their deliberations. Despite these theoretical possibilities of negative impact, the team ran across only a few examples. In view of the high percentage of participants who affirmed impact, the team did not feel it necessary to seek out examples of negative impact. Modifications to the Scope of Work The original Scope of Work ("old SOW") for the assessment included in the ATLAS project paper in 1990 established objectives too far-reaching and included a built-in bias, which led to its revision in the SOW for this assessment. The original stated the following: The purpose of the Impact Study is to establish a body of evidence that USAID￾sponsored U.S. academic training has been critical to the development process and has made direct contributions to economic and social growth. The evidence should be expressed in verifiable and quantitatively specific terms including employment creation, investment earnings, productivity, mortality/morbidity, cost benefit ratios, rates of return and multiplier effects. The study would not be concerned with the training process; it would deal directly with the quality and relevance of the training content, effects on the participant’s career, and the impact resulting from utilizing the training. Recognizing that resources were limited, the SOW written in 2003 for this impact assessment ("new SOW") trimmed down these original expectations. The bias in the old SOW (to establish... evidence... that training has been critical...) also had to be expunged since the critical question to answer from the evidence was whether impact occurred, not to find evidence that it did. The new SOW (included in the Annexes) called for assessing "how well the ATLAS Project program indicators met its purposes and goals." The assessment team decided, with the concurrence of the CTO, not to follow this guidance rigidly, which would have sent the team back to conduct a narrow, traditional assessment to discover whether the project met its anticipated outcomes. With the original goals and indicators in mind, the team designed an "impact assessment framework" to zero in on those indicators that could reveal characteristics about change. For example, "employment" (from the list above) was questioned as a useful indicator since counting the number of participants occupying civil service positions, for example, yields little information about impact. On the other hand, the "level of authority" indicator could imply at least the possibility that the participant effected change. The methodology included ways to ensure that female participants were selected, interviewed and included in the data gathering to be able to compare male with female "performance," which the team interpreted to mean "impact." The resulting participant questionnaire gathers data respecting these indicators and framework. The ATLAS project targeted African institutions, rather than qualified individuals, for capacity building. One of the hallmarks of the AFGRAD project was its commitment not to continue the brain-drain that had been criticized in the predecessor ASPAU program, which was terminated in the 1970s, after evaluators disclosed a 35 percent "repatriation" rate.5 The focus on institutions helped the team rely on an impact assessment framework. The new SOW tasked the team to visit four U.S. universities that participated in the program. In order to focus more on obtaining evidence of impact from the field, the team recommended that this task in the SOW be dropped. This decision freed up resources that were used, for example, to add a seventh country (Francophone) to the site visits. 5 African Scholarship Program for American Universities, one of the early USAID-funded programs that AAI administered in the 1960s for undergraduate African students at the U.S. colleges and universities. ANNEX C THE MODIFIED KIRKPATRICK FRAMEWORK ______________________________________________________________________________ A methodological framework is desirable in conducting any evaluation in order to maintain focus on the objectives and enhance the analysis of data. The survey instruments must be conceived in relation to the framework in order to avoid unnecessary (and costly) data collection or confusion in the collection of the data itself. All too often survey instruments are a mish-mash of questions that have no internal cohesion or logic, reflecting unclear evaluation objectives or the lack of a guiding framework. Donald Kirkpatrick's Four Levels of Evaluation, created in 1959, allows assessment teams to view impact from training through a prism that gives structure and soundness to the journey. The time-tested framework is simple yet adaptable to a variety of settings, including developing countries. The evaluation "hierarchy" –with the individual at its base and the institution at its peak – supplies a tool to begin the search for impact from training and allows for add-ons and adaptations along the way. The Kirkpatrick methodology easily guides the development of a concise and targeted survey instrument by enabling the designers to aim questions at four levels. This section briefly describes the framework and the modifications made by the assessment team. The brief description of the four evaluation levels that follows is intended for readers unfamiliar with the approach.6 (1) Reaction - the trainee’s impression of the program; the level of satisfaction with the course, trainer, pace of instruction, content and materials; (2) Learning - the acquisition of skills and knowledge from the training; (3) Application - the performance of the trainee on the job following training; (4) Results - changes that the trainee’s performance brought to the organization in efficiency, productivity or profitability. Kirkpatrick’s Evaluation Model Data gathered at Level One (Reaction) tells us the minimum amount of information needed to assess a training program. If the trainees judged the program “very satisfying,” we can safely eliminate an important obstacle to obtaining results from training: displeasure with the training received. If the trainees were satisfied with the program, did they learn anything? A Level Two (Learning) enquiry helps determine whether skills and/or knowledge were acquired, and/or attitudes changed during the training program. Without this transfer, impact cannot occur. If the transfer is noted, then a performance change could occur if the KSA were applied at the work place. 6 Further information can be found on the Internet at any number of training-related websites (American Society for Training and Development, International Society for Performance Improvement, etc.). The Level Three (Application) question asks that if skills were acquired during training, were they subsequently applied at the work place? Without application, human performance change is hypothetical and KSA remain within the individual. Training is limited to a “feel-good experience” that does not lead to institutional impact but might appear useful to the participant. A Level Four (Results) question seeks to determine whether any performance changes at the organizational level resulted from the application of skills at the work place. This is the highest evaluation level and one infrequently documented in performance assessments. Change attributed to training at Levels 2, 3 or 4 is impact. The diagram below portrays the modified Kirkpatrick approach adopted by this assessment. The left arrows display the essence of his evaluation levels: reaction (or satisfaction), acquisition of KSA – Knowledge, Skills and new Attitudes (or learning), application of KSA at the work place, and results/output (or impact). A trainee is satisfied with the "learning environment" or program (Level 1), learns something new and presumably useful (Level 2), returns to the work place to apply what was learned (Level 3) and after a lapse of time, the institution increases its output as a result of the trainee's application of what was learned (Level 4). In a nutshell, this is Kirkpatrick's common-sense contribution to evaluating the impact from training that is used to this day in U.S. industries and world-wide. Evaluations conducted by USAID for years considered primarily Levels 1 and 2. But in the 1990s, as discussed in Section B, USAID developed a framework drawing from Kirkpatrick and others within which impact could be better analyzed and understood beyond the individual. Definitions abound about what these terms mean, but fundamentally, they adapt easily to diverse environments. Kirkpatrick's legacy was a de-emphasis of the individual as a target to measure changes attributable to training, in favor of the application of the benefits of training on organizational output. But to use the framework in the African setting, and in particular to assess impact from long-term training, the team added impact "venues" (sector, community, region, nation, international) to capture impact outside an institution – in fact, far beyond a single organization. In the course of designing a survey instrument, the assessment team decided to expand the areas where impact might occur, as described in the new SOW, to include community and international. The former was not carefully defined, the thought being that some participants (through NGOs they created, for example) might not have a sectoral or national impact, but could introduce changes in the community. The latter was added to capture impact that might be identified beyond Africa, for example, in international organizations where participants are employed. This assessment employs the modified Kirkpatrick framework above to determine whether impact occurred both at the institutional level and beyond. Added is the top "level" to take into account impact that may occur beyond institutional boundaries, for instance, in a sector, or at the national, regional or international level. In view of the program's emphasis on academic training, the team decided to employ Kirkpatrick's evaluation levels as a useful framework to guide the assessment. For over 50 years the "method of choice" for training evaluations for private sector clients has been Donald Kirkpatrick's Four Levels, or the many variations thereof. The application of Kirkpatrick's approach by developing country evaluators to public-sector clients, or to the not-for-profit sector, Kirkpatrick's 4 Evaluation Levels* Results/Output Application Acquisition Institutional Individual Sectoral, National, Regional & Int'l Reaction Individual Impact Levels Modified Results/Output Institutional *The top (5th) level has been added was conceived and systemized in the 1990s under USAID's HRDA project (Human Resources Development Assistance) implemented in Africa.7 7 The HRDA project (implemented by AMEX International and Creative Associates International) produced the 9- part series Best Practices for Results-Oriented Training primarily for USAID Missions. The Team Leader for this ATLAS/AFGRAD Impact Assessment was one of the principal authors of the HRDA publication. ANNEX D DATA COLLECTION METHODOLOGY AND COUNTRY SELECTION ______________________________________________________________________________ This annex presents the steps followed and details about organizing and administering the participant survey not included in the main report. It also presents the logic behind the selection of seven countries for in-depth anecdotal information collection. The table below summarizes the evolution of the sampling universe from original to "revised." The relative standard error (RSE) percentages apply only to the entire selection below. Each table presented in the body of the report contains RSEs for that particular table to allow the reader to assess the statistical reliability of the data produced from a single cross-tabulation. Building the Sample The steps used to build the sample began with a thorough review of the AAI database that contained records on all the participants from the 45 countries through the formulation of randomly-selected country and participant lists. After 27 countries were randomly selected, 671 participants were randomly-selected from these 27 countries as "core" participants to locate for * In the Original (from AAI) and Revised Universes, there were 342 and 205 cases respectively where participant gender was not entered. The 205 have been distributed in proportion to the reported cases. Table 1: Characteristics of the Universe and the Sample Original Universe Revised Universe Unweighted Sample Weighted Estimate Category Number Percent Number Percent Number Percent Number Percent Relative Standard Error % All participants 3,219 100.0 1,921 100.0 203 100.0 1,921 100.0 By language group English 1,705 53.0 1,126 58.6 100 49.3 1,154 60.1 5.5 French 1,321 41.0 657 34.2 72 35.5 629 32.7 9.7 Portuguese 193 6.0 138 7.2 31 15.3 138 7.2 33.3 By gender Male 2,578 80.1 1,491 77.6 136 67.0 1,262 65.7 4.8 Female 641 19.9 430 22.4 67 33.0 659 34.3 9.1 By program Afgrad 2,455 76.3 1,359 70.7 97 47.8 1,187 61.8 5.4 Atlas 764 23.7 562 29.3 106 52.2 734 38.2 8.7 No gender reported* 342 10.6 205 10.7 the survey. After administering the survey, to accommodate changes encountered, the "original survey universe" was modified, as shown under "revised survey universe." The weights were then applied to the sample as well as the relative standard error calculations. The approach contained the following steps. • The AAI database had to be analyzed, verified, corrected and transferred to a usable software program. • The team selected the principal stratifying factors to be used on the eventual sampling: gender, geographic area and the program (ATLAS/AFGRAD). • The target for the overall standard deviation was discussed and put at approximately 10 percent out of a universe of 3,219 participants, which would produce an estimated 2,360 names. • The size of the sampling was put at roughly 14 percent, or roughly one out of seven participants in each of the selected countries • A 2-stage sample was created, without affecting the standard deviation, to spread the sample geographically as much as possible to include as many countries as resources would allow thereby reducing the number of respondents to locate and survey in each selected country. • In the first stage, a random selection of 27 countries from the 45 was undertaken using a unique stratification program developed by the Bureau of the Census to help survey designers find minimum-cost, stratified samples that satisfy multiple variance constraints. This first "cut" was appropriate because the variation between countries was greater than the variation within each country • In the second stage, individual participants within each selected country were randomly selected. • Due to lack of accurate participant contact information in the AAI database, the team over sampled by double the number of targeted participants for each country to arrive at a total of 671 randomly selected participants, of which the team would attempt to survey half (335). • Intra-country weights would then be derived by dividing the total number of participants by the total number of completed questionnaires, as shown in the chart below. Table 1: Characteristics of Universe and Sample Original Universe Revised Universe Country Total # of part. Sample Status # of selected part Status # of selected part. Country weight Benin 96 Not selected Included 19 5.0526 Botswana 31 Not selected 0 0 Burkina Faso 46 Selected 13 Included 4 11.5000 Burundi 40 Not selected 0 0 Cameroon 126 Not selected 0 0 Cape Verde 37 Selected 11 0 Central African Rep 25 Not selected 0 0 Chad 10 Not selected 0 0 Comoros 10 Not selected 0 0 Congo, Dem. Rep. of 119 Selected 34 Included 1 (b) 44.5000 Congo, Republic of 48 Not selected 0 0 Cote D’Ivoire 116 Not selected 0 0 Djibouti 6 Selected 2 0 Equatorial Guinea 7 Not selected 0 0 Eritrea 26 Not selected 0 0 Ethiopia 132 Selected 38 0 Gambia 25 Not selected 0 0 Ghana 215 Selected 61 Included 32 6.7188 Guinea 96 Not selected 0 0 Guinea-Bissau 57 Selected 16 Included 3 19.0000 Kenya 125 Selected 35 0 Lesotho 40 Not selected 0 0 Liberia 83 Selected 24 Included 1 (a) 60.6000 Madagascar 100 Selected 28 Included 15 6.6667 Malawi 86 Selected 24 Included 4 21.5000 Mali 115 Selected 33 Included 24 4.7917 Mauritania 29 Selected 8 Included 2 (b) 44.5000 Mauritius 28 Selected 8 Included 1 (a) 60.6000 Mozambique 81 Selected 23 Included 28 2.8929 Namibia 77 Selected 22 Included 31 2.4839 Niger 44 Not selected 0 0 Nigeria 203 Selected 58 Included 3 67.6667 Rwanda 30 Selected 9 Included 1 (b) 44.5000 Sao Tome & Principe 11 Not selected 0 0 Senegal 94 Selected 27 Included 6 15.6667 Sierra Leone 96 Selected 28 Included 1 (a) 60.6000 Somalia 49 Selected 14 0 South Africa 36 Selected 10 Included 1 (a) 60.6000 Table 1: Characteristics of Universe and Sample Original Universe Revised Universe Sudan 78 Selected 22 0 Swaziland 60 Selected 17 Included 1 (a) 60.6000 Tanzania 142 Selected 41 Included 3 47.3333 Togo 94 Not selected 0 0 Uganda 128 Selected 37 Included 22 5.8182 Zambia 98 Selected 28 0 Zimbabwe 24 Not selected 0 0 Totals 45 countries 21 countries 3,219 part. 671 1,921 total participants 203 questionnaires (a) = Combined English (b) = Combined French Administering the Survey Even when reached via the Internet or telephone, many participants did not return completed questionnaires. What were the reasons for such unanticipated difficulty locating participants from the 20 "non-visited" countries, or persuading them to complete and return the questionnaire? Was their an unwillingness to cooperate, was the team's approach ineffective, or were participants too busy? The team arrived at the following possible reasons: • Poor Contact Data: Although the database kept by AAI was full of information on each participant (name of employer, date of beginning and completion of training, country, sector, degree, subsequent degree, etc.), the records lacked current contact information.8 Many phone numbers no longer worked or required new prefixes as countries updated local exchanges throughout the 1990s. Of the few email addresses in the database, many bounced back. Efforts to rectify the situation were unsuccessful within the time and resources the team had available. In some cases, AAI reported that its former local country representatives had access to updated information, but the team's experience in the field did not support this view. • Security Concerns: For the following countries, the team had particular difficulty locating participants, despite many attempts through informal and formal U.S. networks: Sudan, Ethiopia, Liberia and Sierra Leone. The reasons vary: for Sudan, the team surmised and then confirmed that security concerns made it nearly impossible to penetrate the U.S. networks. For Liberia and Sierra Leone, the same difficulties were encountered in terms of eliciting help from informal networks, but perhaps the reasons were more related to suspicions about sharing information among these nationals themselves residing in the 8 For unknown reasons, participant gender was missing from roughly 10 percent of the entries, even after AAI submitted corrections. As a partial solution, the team was tasked to assign "Male" or "Female" to participant records when certain the first (i.e., "given" or "non-family") name indicated the gender. This left roughly 200 names whose genders could not be recognized that were then distributed in relation to their respective proportion of the total. U.S. It was thought that Ethiopia would be easy, but the team was unsuccessful either in obtaining assistance from the U.S. network or, significantly, from the USAID Mission in Addis Ababa. Efforts to identify foreign nationals working at U.S.-based international organizations were largely unsuccessful, primarily due to post 9/11 security issues. • Suspicions: The participant meetings revealed that there had been little effort by USAID over the years to maintain contact (follow-up) with returned participants. Why, then, was it undertaking a large impact assessment at this point? The team guessed that this might have been a reason for some participants to be lax or uninterested in completing questionnaires. • Self-interest: Some asked directly what they would "get" by completing the questionnaire (which would require roughly 40 minutes). At participant meetings, team members were asked why USAID funding for long-term U.S. training had declined (or disappeared, in the case of Ghana and Uganda). If the assessment were not a precursor to increased USAID funding, why bother completing the questionnaire? It must be said that the majority of participants with whom the coordinator was able to speak directly were cooperative and completed the questionnaire. Many of these helped locate others. One woman participant, hearing of the survey, made herself known, downloaded a questionnaire, completed it by hand, and expedited it by DHL at a cost of over $100 to Washington! In Uganda, a number of participants missed the participant meeting and insisted on being able to complete the questionnaire. Weeks after the deadline, questionnaires would show up, occasionally from someone the team had not contacted but whose name was on the core list. The team's response to this unexpectedly-low return rate was to develop an abbreviated version of the longer questionnaire that could be administered by telephone. Key questions were taken from the original questionnaire and asked verbally adhering exactly to the text as written. The integrity of the longer survey instrument was maintained by carefully reading the questions without comment or assistance. This corrective course proved successful and the team increased the number of questionnaires from 172 to 203. To facilitate data entry from questionnaires written by African professionals and to maximize the use of resources, hard copies of all completed questionnaires were sent to a Ghanaian consulting company experienced in managing survey data. Trained staff developed the SPSS template and entered into the system the quantitative answers on 203 questionnaires collected, after which the data record files were sent electronically to the team's statistical specialist in Washington. Cross￾tabulations were then run of selected questions on the questionnaire. Since the questionnaire contained valuable open-ended answers written by respondents, those text answers were typed into an Excel spreadsheet in Ghana and electronically sent to the Team Leader for analysis. Many of the text entries validated participants' answers to "yes/no" questions. This process, known as "grounding," increases the credibility of participant answers by forcing them to provide concrete examples to justify their claim. Other text entries contained comments and suggestions that could be reviewed to get insights into impact, or from which to draw lessons for future long-term training. Because these open-ended entries were organized correctly, they could be formatted and printed to facilitate rapid reading and analysis. (These records of participants' own text answers are included in the Annexes to provide readers with an opportunity to view directly what participants wrote.) In some cases, the analysis of the text answers was quantitative. For example, when participants wrote similar or identical answers to questions, they were grouped during the analysis, and then arranged according to the frequency with which they were cited. The results from this type of analysis are included in several sections of the report and provide a quasi-quantitative analysis of answers to open-ended questions. (The list of text answers to many open-ended questions is included in the Volume III Annexes and offers the reader an unusual glimpse into participants' professional lives and accomplishments.) Revising the Survey Sample Despite some success at rectifying the problems discussed above that impeded collection of questionnaires, the team decided to terminate further efforts and devote resources to compilation of the data and analysis. Eventually 203 completed questionnaires from twenty-one (21) countries comprised the sample, although the representation of non-site-visit countries is limited. The distribution of completed questionnaires by country is shown in the table in the Annexes. Because the response was so limited, the universe definition had to be revised. The universe was re-defined to include only those countries (21) with one or more returned questionnaires, instead of the representative random sampling of all 45 countries in the program. The following countries were removed due to the low number of questionnaires received: Cape Verde, Djibouti, Ethiopia, Kenya, Somalia, Sudan, and Zambia. Therefore, the survey results apply only to this more limited universe of 21 countries rather than to a "representative sampling" of the 45 countries, as shown in the table Final Sample by Country below. This revised universe includes 1,921 participants or approximately 60% of the 3,219 participants in the original universe definition, slightly less than the original target of 2,230. Weights were assigned for each country by a simple ratio of the returned questionnaires divided by the total number of participants in the program. With these adjustments, it was no longer possible to calculate relative standard errors for countries with fewer than three returned questionnaires. Therefore, those countries were grouped by language as follows: • Countries with English as the official language: Liberia, Mauritius, Sierra Leone, South Africa, and Swaziland • Countries with French as the official language: Democratic Republic of Congo, Mauritania and Rwanda What are the consequences of the changes described above to the original participant survey that were instituted to overcome the data collection constraints? The principal change is that the assessment cannot extrapolate quantifiable data from the survey of 21 countries, to the 45 in the ATLAS/AFGRAD universe as had been anticipated. But the assessment can speak authoritatively about the 21 countries from which quantitative data has been culled with relatively low standards of error, and following statistical norms. Furthermore, the survey instrument contains extensive qualitative data that contributes as much to the credibility of the findings as the quantitative data, as is shown in the next sections. In fact, there are quantitative elements of the text answers that participants submitted in that their answers were grouped according to similarity and analyzed accordingly. Table 2: Final Sample by Country Original Universe Revised Universe Country Total # of part. Sample Status # of selected part Status # of selected part. Country weight Benin 96 Not selected Included 19 5.0526 Burkina Faso 46 Selected 13 Included 4 11.5000 Cape Verde 37 Selected 11 0 Congo, Dem. Rep. of 119 Selected 34 Included 1 (b) 44.5000 Djibouti 6 Selected 2 0 Ethiopia 132 Selected 38 0 Ghana 215 Selected 61 Included 32 6.7188 Guinea-Bissau 57 Selected 16 Included 3 19.0000 Kenya 125 Selected 35 0 Liberia 83 Selected 24 Included 1 (a) 60.6000 Madagascar 100 Selected 28 Included 15 6.6667 Malawi 86 Selected 24 Included 4 21.5000 Mali 115 Selected 33 Included 24 4.7917 Mauritania 29 Selected 8 Included 2 (b) 44.5000 Mauritius 28 Selected 8 Included 1 (a) 60.6000 Mozambique 81 Selected 23 Included 28 2.8929 Namibia 77 Selected 22 Included 31 2.4839 Nigeria 203 Selected 58 Included 3 67.6667 Rwanda 30 Selected 9 Included 1 (b) 44.5000 Senegal 94 Selected 27 Included 6 15.6667 Sierra Leone 96 Selected 28 Included 1 (a) 60.6000 Somalia 49 Selected 14 0 South Africa 36 Selected 10 Included 1 (a) 60.6000 Sudan 78 Selected 22 0 Swaziland 60 Selected 17 Included 1 (a) 60.6000 Tanzania 142 Selected 41 Included 3 47.3333 Uganda 128 Selected 37 Included 22 5.8182 Zambia 98 Selected 28 0 Totals 3,219 27 countries 671 21 countries 1,921* . *weighted 203 questionnaires (a) = Combined English (b) = Combined French Developing the Questionnaire A sample questionnaire was developed following the team's modified Kirkpatrick framework that conceived impact as accruing from the individual's acquisition of knowledge, skills and new attitudes through application, institutional change and extended change (sector, region, etc.). At each level, with the exception of Level 1 (Satisfaction), the survey instrument sought participants' views on impact. Questions at Level 1 (such as, "Did you like your academic program" or "How would you rate the quality of the training?") were de-emphasized in this assessment. Such questions dominate many training assessment evaluations and produce little information regarding impact. The questionnaire was divided into the following sections: • Information before training began (employment, organizational type and sector, etc.) • Information about the training program (degree level obtained, follow-up activities, etc.) • Information after returning (employment, positions held, KSA acquired and applied, institutional changes, participant impact beyond the institution, overall views, etc.) Once drafted, the questionnaire was shared with outside specialists knowledgeable about surveys and evaluations, revised, tested with four ATLAS/AFGRAD participants, revised again and finalized. Administering the Questionnaire and Organizing the Data The Master List contained 671 names of participants in the 7 "site-visit" countries9 and 20 "non￾visit" countries. For the 20 non-visit countries, the assessment contractor, Aguirre International, assisted the team by hiring and overseeing a bi-lingual coordinator familiar with Africa whose sole task was to search for participants on the core list. For the participants from countries to be visited, the local coordinators hired to set up the ATLAS/AFGRAD participant meetings were tasked with identifying participants from the core list as well as promoting the participant meeting to all returned alumni. The objective was to first locate the participants (by telephone and email), then ask them to complete a detailed questionnaire and to return it by email or fax. The lack of reliable contact information from AAI significantly affected the team's allocation of resources. What was expected to be a 3-week process of locating participants consumed months of effort. The table reflects the gravity of the problem: an average of fully 26 percent of the records had no contact addresses at all – that is, nothing entered in the data field. The table also shows that the lack of contact information was not closely linked to their year of training, except for the most recent entries: even for participants from 1990 to 1994, over one-fourth had no contact information at all. For the remaining 74 percent that had some data entered, the team discovered serious problems with an estimated 75 percent of the participants in the random pool of 671 sample names. By June 22, 2004, using every possible source and method, the team had located only 156 participants, or 46 percent of its target of 335 (half of the over sampling of 671), of which 47 were from the 7 site-visit countries (where local coordinators did not rely on the AAI database information). Of these 156 "hits" (that is, participants located), only 83 questionnaires had been received, or 26 percent of the requirement. To make matters more difficult, the distribution was uneven, with some countries having no "located" participants. On the other hand, other ATLAS/AFGRAD participants from the site-visit countries, whose names were not randomly selected as "core," did submit completed questionnaires, usually during participant meetings in￾country. Questionnaires from both sub-sets totaled 172. Eventually 203 completed questionnaires from twenty-one (21) countries comprised the revised sample. he following countries were removed due to the low number of questionnaires received: Cape Verde, Djibouti, Ethiopia, Kenya, Somalia, Sudan, and Zambia. 9 See the next section for a description of the selection of the "site-visit" countries. Number of AAI Records with No Contact Address Year Group No. of Alumni Alumni with no Address Percent no Address Before 1970 193 58 30.1% 1970-1974 385 111 28.8% 1975-1979 439 123 28.0% 1980-1984 488 135 27.7% 1985-1989 368 112 30.4% 1990-1994 451 118 26.2% 1995-1999 553 92 16.6% 2000-2002 172 14 8.1% Withdrew 200 92 46.0% Unknown/blank 14 8 57.1% Total 3263 863 26.4% Country Site Visits This section will address the following activities related to the site visits the team made to the seven countries: Benin, Ghana, Mali, Madagascar, Namibia, and Uganda. • Selection of the countries visited • Participant meetings • Institutional visits • Individual participant interviews This set of countries was balanced by program (as shown above), language and region. Methodology for the Selection of Site Visit Countries The team analyzed each of the 45 sub-Saharan African countries and grouped them first by elimination. A short-list was arrived at by removing from consideration most of the countries for the reasons shown below. • No USAID Mission and sampling or participant numbers too small: Sao Tome Chad Equatorial Guinea Comoros Djibouti Gambia CAR Mauritania Lesotho Swaziland Cape Verde • Security issues precluded an effective team visit: Liberia Sudan Kenya Zimbabwe Cote d'Ivoire Somalia Burundi • Security was not a major constraint but the risk of not finding sufficient numbers of returned participants existed due to recent civil unrest or a fragile political environment; moreover, or there was no USAID Mission: Congo (Brazzaville) Swaziland Burkina Faso Guinea-Bissau Sierra Leone Togo • Program too small to justify allocating resources for a visit, although there was a USAID Mission: Botswana Eritrea Rwanda Niger The following criteria were also applied. • Region: as many regions as possible should be represented. • Language: at least one Lusophone, and a reasonable balance between francophone and Anglophone, considering the participant language split below: Anglophone 53% Francophone 41% Lusophone 6% • USAID Mission: presence of a Mission was considered essential for a site visit. Although the Mission was not involved in any local arrangements, the team sought evidence of an interest in the assessment's outcome. In fact, in every country visited, USAID representatives attended part or the entire returned participant meeting. In most cases, Missions also met with the team. • Program: a selection of countries that had participants from both ATLAS and AFGRAD programs was important. It was not necessary that every country selected have such a mix (which would have excluded some critical countries, such as Ghana and Namibia) but that the total group of countries visited include participants from both programs. The split between these programs in the database was: ATLAS 24% AFGRAD 76% Despite the relatively small portion represented by ATLAS, the team thought to balance the two more equally in terms of country selection because ATLAS was more recent and contained certain program innovations not found in its predecessor. The final split in participants for each proposed country are shown below. • Efficiency: As with any assessment, there were limited time and resources available that had to be factored into the final selection. After applying the criteria described above, the following short list was established: Short List Francophone West Africa Senegal Benin Mali Guinea Anglophone West Africa Ghana Francophone East Africa none Anglophone East Africa Tanzania Uganda Ethiopia Central Africa Cameroon (Fr/Ang) Anglophone Southern Africa Zambia Malawi Namibia Francophone Southern Africa Madagascar Lusophone Africa Mozambique The team analyzed the pros and cons within each of the above groupings to arrive at the final list of seven countries below. The reasons behind the final decisions are discussed in detail in a memo included in the Volume II Annexes (Memo of December 14, 2003). Team members were assigned to each country according to experience and language fluency. After the first two site visits were completed (Mozambique and Namibia), the team met in Africa at a convenient crossroads (Senegal) to conduct a thorough review to ensure that the information being sought was obtained, and to make mid-course adjustments for the remaining five site visits. As a result of this 2-day meeting, the team made several critical modifications that improved subsequent information-gathering and, most importantly, assured consistency in the way impact was being assessed in each country. Participant Meetings Although the Scope called for organizing a participant "focus group," the team decided to reformulate that crucial information-gathering mechanism. Instead of an intimate, more informal meeting of "key informants" in the style of a "focus group," a larger, more formal "participant meeting" would be organized. This would allow the team to challenge a group of perhaps 15 to 25 participants to explore what impact they might have had together, in small work groups, to produce more tangible findings than the outcome from a focus group. This decision was fortuitous in that the facilitated workshop setting produced evidence-based findings on impact at various levels. After the team presenters led an interactive discussion of the assessment's objective, methodology employed and definition of impact, small work groups were formed and instructed to a) write down a description of impact, if any, each participant believed to have had that was attributed to the training received, b) present one-by-one these statements to the small group, and c) select one to present to the others in the plenary session. The result was a collection of hand￾written statements from each U.S.-educated participant that described precisely, often many years after returning home, the changes that occurred linked to the ATLAS/AFGRAD program. These statements are invaluable glimpses into impact perceived by the trainees themselves that reveal the challenges, frustrations as well as the successes they encountered. Final Site Visit Selection with Program Percentages AFGRAD ATLAS % % Benin 35 65 Ghana 100 0 Mali 70 30 Madagascar 50 50 Mozambique 75 25 Namibia 0 100 Uganda 80 20 Total Averaged 58 42 Participant meetings were organized by local coordinators hired to publicize the upcoming meeting, locate participants, distribute questionnaires to be returned or completed at the meeting and arrange local appointments for the team members. To ensure objectivity in the assessment's field work, the team decided early-on not to rely on former or current AAI representatives in the countries to be visited, with the sole exception of Mozambique (where no available alternative could be found). Local coordinators ranged from former participants working as independent consultants to local consulting firms specializing in human resource and training. In each case the local coordinators informed the former AAI representatives of the assessment and invited their help in locating participants, but did not involve them in the local program. Institutional Visits and Follow-up Participant Interviews In each country, team members visited local institutions to meet with decision-makers who might be able to comment on whether changes occurred after a participant's return that could be attributable to long-term training. In some cases, the team was able to interview a supervisor who could give impressions of performance before and after receiving the training, a rare opportunity in the fast-changing organizational context of African institutions. Team members were able in some cases to visit laboratories or research stations that were begun by former participants. Based on the first two site visits to Mozambique and Namibia, the team decided to select in each country one or two outstanding alumni with extraordinary stories about changes they attributed to their U.S. training. The objective was to probe more deeply into the characteristics of this impact and the obstacles the participant overcame. After each meeting, the team identified these participants and set up follow-up appointments. Team members produced short descriptions of these "special impact stories" for the record. Participants Attending Country Meetings Benin 19 Ghana 28 Madagascar 10 Mali 16 Mozambique 24 Namibia 25 Uganda 14 Total 136 ANNEX E ABBREVIATED QUICK-REFERENCE VERSION OF QUESTIONNAIRE ______________________________________________________________________________ Note: Most check boxes have been removed from the answers (for example, "Yes/No" and other multiple choices) to facilitate quick reference to the questions when reading the report. Please consult the complete questionnaire in the Annexes for the unchanged version. 1. Were you employed by an organization when you left for training? 2. If “No,” were you: Self-employed , Student , Unemployed- looking for work, Not working and not looking for work (for ex., due to family reasons) Other (05) 3. Which of the following best describes the kind of organization you worked for? Private for-profit sector (business) , Private non-profit organization (NGO) , public sector (government, parastatal), Donor organization (USAID, UNDP, World Bank, etc), Other (please describe): Not applicable 4. In what sector did the organization focus its work (for example, agriculture, health, education, financial, transportation, and management/administration)? 5. What was your job title before you left for training? __________________________ 6. What was your main activity in that job? (Project management, research, teaching, administration, human resources/personnel, other) 7. How long had you worked at this organization before leaving for the United States for training? 8. How were you selected for study in the United States? 9. Once you were selected, how did you choose your academic degree program (select one below)? 10 Was an objective defined for your U.S. program? 11. If “Yes,” please state what the objective was: 12. Was there any assessment (training needs, performance or capacity gap, etc.) conducted at your employer that was linked to your going to the U.S. for study? 13. If “Yes,” please explain: 14. Which USAID-funded program sponsored your training? 15. Did you have to learn English before beginning your academic training? 16. In what year did you first go to the United States to begin your program? ________ 17. Did you receive a degree (diploma) after completing your academic program? 18. If yes, please indicate the highest degree and field below obtained that was funded by ATLAS/AFGRAD or another USAID project, either partially or fully: 19. Name the FIELD or SPECIALTY that you studied in: 20. Did you receive any other degree from the United States, not sponsored by USAID or by ATLAS/AFGRAD (either partially or fully? 21. If yes, please indicate the highest degree and field below. 22. In what year did you receive the highest diploma indicated above? __________ 23. Please indicate how this degree was funded: 24. ATLAS organized seminars and conferences (called “Professional Enhancement Activities”) in the 1990s for both ATLAS and AFGRAD participants. Did you attend any of these? 25. If you attended any of the Professional Enhancement Activities, please indicate your opinion on their usefulness to your professional development: Very useful, fairly useful, somewhat useful, not useful. 26. If you selected “Very Useful,” please explain your selection: 27. AAI also provided post-graduate grants to selected participants, sometimes called "POSTAF" awards. Did you receive one? 28. AAI also made annual awards, called "Distinguished Alumni Awards." Did you receive one? 29. Please select below the response that best represents your opinion: "In relation to my professional development, I believe that the contribution that my ATLAS/AGRAD made to my professional development was: Very important, important, neither/neutral, not very important, not at all important no opinion 30. In your view, how much has your U.S. degree affected your career path? A great deal, a lot, some, a little, not at all 31. Since your return from the U.S. after your ATLAS or AFGRAD experience, have you worked for the same institution where you worked prior to leaving your country? 32. If “No,” please indicate the NUMBER of institutions you have worked for since your return? 33. What kind of organization do you work in currently? 34. Have your responsibilities increased, stayed the same, or decreased since your return from training? Increased, stayed the same, decreased? 35. Please explain your choice: 36. Did you acquire any specific knowledge, skills, or new attitudes from your academic program in the United States? 37. If “Yes,” please indicate examples of the three most important skills, attitudes, or specific knowledge that you acquired from your U.S. academic program. 38. If you answered "yes" to the question above, how much of the knowledge, skills or new attitudes (from your list above) did you apply in your work? A great deal, a lot, some, a little, none, no opinion 39. If you selected "A lot" or "A great deal," please indicate which ones you applied. 40. If you selected "None," A little" or "Some," please explain your selection. 41. Please indicate how difficult or easy it was to apply your knowledge, skills and new attitudes at the institution where you worked immediately after returning: very easy, fairly easy, possible but difficult, very difficult, impossible 42. In the course of applying what you learned in the U.S. to your work, were you able to share your knowledge, skills, and new attitudes with others? 43. How supportive were those people who supervised your work when you applied your newly-acquired knowledge? Very supportive, somewhat supportive, neither supportive nor resistant, somewhat resistant; No opinion? 44. Please explain your answer. 45. Have you been able to apply the knowledge, skills or new attitudes acquired in your U.S. experience to other areas of your life, such as within your family, in other professional relationships, or in the community? 46. If “Yes,” please give some examples of how you have done so. 47. If you were able to apply your knowledge, skills or new attitudes at work (in other words, you answered "Yes" to Question 0), has there been any difference in your institution's output (in terms of quality, quantity, etc.), performance, productivity or impact? In other words, did anything change? 48. If “Yes,” please give one or two concrete examples below: 49. If you answered "Yes" to Question 47 above, please indicate below at what level the change you describe took place (please select all that apply).  a) within the institution where you worked  b) in the sector targeted by the institution where you worked  c) in the community  d) nationally  e) regionally (such as West Africa)  e) regionally (such as West Africa) 50. Please explain your answer or add any relevant information to help us understand at what level you were able to bring about change. 51. Are there any specific achievements or discoveries or contributions that you have made that are directly linked, in your view, to the knowledge, skill and new attitudes that you said you acquired in the United States? 52. If “Yes,” please tell us about the most important ones below: 53. Consider your present income and economic circumstances as compared with all the people in your home country. Where would you place yourself in terms of your present economic circumstances? Bottom 50%, top 50%, top 20%, top 10%, top 5%. 54. To what degree to you attribute your present income (or economic circumstances) to your participation in the AFGRAD or ATLAS program? (Totally/very, a great deal, somewhat, not much, not at all.) 55. Have you been involved in the creation, expansion, or ongoing management of a private for-profit company or companies? 56. If “Yes,” please estimate the number of jobs that have been created as a result of your participation. ______ 57. Do you know of any other ATLAS or AFGRAD returned participant who has made specific achievements or discoveries or contributions that may be related to their U.S. training? 58. If “Yes,” please tell us about them (the contribution made and the person's name): 59. Have you ever received an ATLAS/AFGRAD Alumni Directory? 60. If yes, in what year did you receive the last Directory? _________ 61. If yes, please check below how you used the directory. 62. Since your return, have you ever attended an "alumni" meeting of former U.S. participants (not just from ATLAS/AFGRAD, but from any US Government program)? 63. Does such an association exist where you reside now? 64. If “Yes,” is it active? 65. If “Yes,” is it useful to the development of your professional life? 66. Please explain how the association has contributed to your professional life. 67. Do you interact in your professional life with colleagues who were graduated from universities in other non-North American, non-African countries, such as the U.K., France, China, Belgium, the former Soviet Union, Germany, etc.? 68. If “Yes,” please check any listed below where you believe there are differences between the two groups (that is, those educated in the U.S. and those in other non-North American, non-African countries) and explain your answers in the space provided. a) Career advancement b) Professional curiosity or commitment c) Access to professional information d) Consulting opportunities e) Reputation at home f) Ability to network with former professors g) Research opportunities h) Other Please rate the different training “types” below in relation to their possible contribution to the social and economic development of your country. Indicate your rating on a scale of 1 to 5, with 1 being “least effective” and 5 being “most effective.” 69. Long -Term training residing in the U.S. 70. Short-term training in the U.S. 71. Distance education obtained over the Internet without long-term residency in the US: 72. Short-term training in my home country 73. A combination of short-term training in my country and distance education, with occasional short-term visits to a U.S. campus 74. The AFGRAD and ATLAS Programs aspired to see alumni contributing to economic and social development in their home countries and in the region. Please check all the following statements that apply to you, as you reflect on your career path and achievements, and give us a brief explanation of why you checked it. (a) I was able to contribute to political and/or economic reforms in my country. (b) I have held appointed or elective office in my country’s government. (c) I have worked in specific economic or social development projects. (d) I have been able to contribute to scientific research and/or application. (e) I have held policy-making positions in government or the private sector. (f) I have supported the development of democratic practices and/or institutions Do you have any specific recommendations or general comments you would like to make in relation to this Impact Study? If so, please write them in the space below. ANNEX F QUALIFICATIONS OF THE DATA by Robert Torene ATLAS Assessment Team Data Analyst ______________________________________________________________________________ The estimates developed from the sample are apt to differ somewhat from the results of a survey covering all participants in the sample lists but otherwise conducted under essentially the same conditions as the actual sample survey. The estimates of the magnitude of the sampling errors (the difference between the estimates obtained and the results theoretically obtained from a comparable, complete-coverage survey) are provided by the standard errors of estimates. The particular sample selected for this survey is one of many similar probability samples that, by chance, might have been selected under the same specifications. Each of the possible samples would yield somewhat different sets of results, and the standard errors are measures of the variation of all the possible sample estimates around the theoretically comparable, complete￾coverage values. Estimates of the standard errors have been computed from the sample data in this survey. They are represented in the form of relative standard errors (the standard errors divided by the estimated values to which they refer, calculated in percentage terms). In conjunction with its associated estimate, the relative standard error may be used to define confidence intervals (ranges that would include the comparable, complete coverage value for specified percentages of all the possible samples). The complete-coverage value would be included in the range: • From one standard error below to one standard error above the derived estimate for about two-thirds of all possible samples. • From two standard errors below to two standard errors above the derived estimate for about 19 out of 20 of all possible samples. • From three standard errors below to three standard errors above the derived estimate for nearly all samples. An inference that the comparable, complete-survey result would be within the indicated ranges would be correct in approximately the relative frequencies shown. Those proportions, therefore, may be interpreted as defining the confidence that the estimates from a particular sample would differ from complete-coverage results by as much as one, two, or three standard errors, respectively. This can be made clearer with several examples. Example 1: Suppose an estimated proportion is shown at 50% with an associated relative standard error of 2 %. One relative standard error interval, therefore, is 1% (50% multiplied by 2%). There is approximately… • 67 percent confidence that the interval 49% to 51% includes the complete coverage total; • 95 percent confidence that the interval 48% to 52% includes the complete-coverage total, and • almost certain confidence that the interval 47% to 53% includes the complete￾coverage total. Example 2: Suppose an estimated proportion is shown at 30% with an associated relative standard error of 6 percent. One relative standard error interval, therefore, is 1.8% (30% multiplied by 6%). There is approximately … • 67 percent confidence that the interval 28.2% to 31.8% includes the complete coverage total; • 95 percent confidence that the interval 26.4% to 33.6% includes the complete￾coverage total; and • almost certain confidence that the interval 24.6% to 35.4% includes the complete￾coverage total. In addition to the sample errors, the estimates are subject to various response and operational errors: errors of collection, reporting, coding, transcription, data entry, etc. These operational errors also would occur if a complete canvass were to be conducted under the same conditions as the survey. Explicit measures of their effects generally are not available. However, it is believed that most of the important operational errors were detected and corrected during the review of the data. The small operational errors usually remain. To some extent, they are compensating in the aggregated totals shown. As derived, the estimated standard errors included part of the effect of the operational errors. The total errors, which depend upon the joint effect of the sampling and operational errors, are usually of the order of size indicated by the standard error, or moderately higher. However, for particular estimates, it is possible that the total error may considerably exceed the standard errors shown. ANNEX G INTERNET IMPACT SEARCH: STRATEGY, FINDINGS AND RECOMMENDATIONS _____________________________________________________________________________ Objectives The goal of this activity was to conduct Internet searches on a random sample of ATLAS/AFGRAD participants to see if such searches could indicate significant contributions made by participants that might escape discovery by other techniques. Finding and Conclusions Although most of the world’s largest and most powerful Internet search tools are located in the U.S or other G8 countries, the indexing and information retrieval powers of Internet search engines and news databases has reached a point where a surprising amount of information can be found, even for citizens of countries which themselves have limited access to the Internet. Key findings and conclusions are noted below: The Internet search found definitive hits on 51 of 100 people sought. Out of a random sample of 100 ATLAS/AFGRAD participants, the researcher was able to find matches for 69. Fifty-one of these “hits”, or fully half of the sample, were clearly identified as participants. Another 18 cases were inconclusive, as information was returned on more than one person with the same name, and the researcher was unable to determine which—if any—were the correct individual. This technique favors certain types of professional profiles, given the nature of the information that is captured in search engines and news databases. Africans who teach at universities, or who work for international organizations like the World Bank, UN agencies, or international NGOs appeared most often. Most of the hits were on published books or academic papers, presentations made at international conferences, directory listings at international or government agencies or universities, or quotes in local media from public events involving the participant. Notably absent were private sector actors, including entrepreneurs who may have started their own companies or otherwise had a significant impact in their country through their economic activities. Unfortunately, this type of search does not capture private sector players—or for that matter civil servants—who work diligently but who do not for whatever reason appear in the public eye or present their work in academic or development agency fora. For those individuals who are captured by this approach, such searches can indicate a participant’s impact, or at least identify numerous useful leads to conduct further research on such potential for impact. Most of the participants found had multiple hits, in some cases as many as several dozen, reflecting a variety of national or international arenas where they were visible and active. In most cases, the researcher had to go beyond the initial search result summaries, clicking on the individual links to properly assess the information on the participant. The Internet researcher did not make a determination on a participant’s “impact,” leaving that decision to the Impact Study Team. However as noted below, the Internet searches did uncover a number of indicators for judging potential impact. This information is captured in a spreadsheet which can be used as the basis for further research on participants of interest to the team. And while they are by no means the only ones within the group that could be considered to have demonstrated impact, a sampling of some notable individuals in the group includes: • Miriam Khamadi Were—an accomplished Kenyan novelist and teacher who later became a medical doctor, university professor, head of a UN office in neighboring Ethiopia, and Chair of the African Medical Research Foundation and of the Kenyan National Aids Control Council • Mulatu Wubneh—an Ethiopian professor of planning and development who has taught at several universities in the US, and who spent an 18-month stint helping set up the Africa Capacity Building Foundation, a joint World Bank, UNDP and African Development Bank initiative. • Linda De Vries—a South African university professor who also started a women’s empowerment investment group, and chairs the Western Cape Gambling and Racing Board • Chiekh Ibrahim Fall—a Senegalese international civil servant who has served as the Vice President and Corporate Secretary of the World Bank, as well as holding a string of high level posts at the African Development Bank Internet searches are a useful research tool to include with other evaluation methods. While this approach should not be used as the sole method to identify a participant and his or her impact, the researcher recommends that such Internet searches be considered as tools in the future to supplement other methods for locating former participants, and for determining the impact they have had in their country or elsewhere. Internet Search Strategy The search strategy was designed to use publicly available Internet resources to seek evidence of ATLAS and AFRGAD participants. One site, AllAfrica.com, charged a nominal fee for access to archival information from prior years. • Beginning with the entire 3,219 participant database, the data analyst excluded all names from countries where surveys are being sent or where focus group sessions are being conducted. • A list of 200 names randomly selected from the remaining participants was provided to the Internet researcher, and served as his Master List. The Master List included participants from 37 Anglophone, Francophone and Lusophone countries. • In order to determine the efficacy of Internet searches across all the countries included in the sample, the researcher alphabetized the list of 200 names by country of origin. He then selected the first participant’s name appearing on each country list for inclusion in the search. In four cases this was the only participant from that country on the list. A second round of participants was selected from 33 countries with two or more participants on the list, and a final set of 30 was selected from countries with three or more participants, for a total Search List of 100 candidates. • The researcher then used seven Internet search tools described below to look for evidence of these 100 candidates. Internet Search Tools—the Internet researcher used a variety of unrelated search tools, including several popular English-language search engines, as well as ones that search and index French, Portuguese and other language sites, and a metasearch tool that combines results produced by other search engines. In addition, searches were made on a database of prominent Africans, and on AllAfrica.com, a content provider which archives news from African print media and international development agencies. Descriptions of the search tools, and why they were chosen for this project, appear at the end of this report. As noted above, the types of references found included: • Research papers, commissioned reports or books written by participants; • Participation in national or international conferences, as evidenced by conference programs, proceedings or lists of participants and/or speakers; • Listings in university or international organization directories; and • News stories quoting or profiling participants’ activities. Matching and Confirmation of Hits—the researcher compared this information with that found in the AFRAD/ATLAS database sample, including addresses, organizational affiliations, academic degrees, etc., to determine if the hit referred to a participant. Where there was a match, a summary of available information was captured on current or previous job titles, papers or books produced, etc., and noted in an Excel spreadsheet Search Results The search results are presented below. (Note: in some cases, the researcher identified and included information on several positions held by an individual over the course of a career, often at different organizations and in different countries. This is why some categories total more than the 51 participants for which information was found. Also note that some information dates back to the early 1990s, so although expressed in the present tense, the statistics may indicate current or previous places of work, job titles, etc.) General Information • Total Participant Pool in Master List: 200 randomly selected names • Participants Researched on Participant List: 100, representing 37 nationalities • Confirmed hits: 51 • Unable to verify hits(s) referred to participant: 18 • No information found on participant: 31 • Gender: Of the 51 hits, 35 were male, 16 were female Place of Work or Residence (does not include US residency for ATLAS/AFGRAD programs) • 18 work/have worked in the US (16) or Canada (2) • 24 work in their home country • 8 work worked in other African countries • 4 unable to determine where they work or live Type of Organization • 17 work for universities (including 7 at US universities) • 12 work for their national government (not including state-run universities) • 10 work for international organizations, primarily the World Bank and UN agencies • 3 work for international NGOs offices in their home countries (Transparency International, Freedom From Hunger, Rodale Institute) • 2 work for national NGOs in the home country • 1 works for USAID in the home country • 1 works in the private sector • 8 unable to determine type of organization Type of Work • 19 University professors or trainers (17 at universities, 1 at donor agency and 1 at national management training institute) • 7 managers in government agencies • 4 managers or professional staff at NGOs or international agencies • 3 work in government-run research organizations • 3 have been Cabinet Ministers • 1 elected Member of Parliament Evidence of High-Level Responsibility and/or Potential Impact In addition to the Cabinet ministers and MP noted above, there also were: • 6 people holding senior management (i.e. Director level and above) positions in government, donor agencies, or NGOs • 3 served as heads of an organization (NGOs, National AIDS Commission, UN agency country office) • 2 were elected officers of Africa regional professional associations • 1 founded a national NGO (women’s empowerment investment group) Other Potential Impact Indicators • 20 authored or co-authored papers or reports, most with international exposure • 8 authored or co-authored books in their areas of expertise (of which 1 is also a popular novelist) • 6 have delivered presentations or papers at international conferences • 2 have delivered presentations or papers at Africa regional conferences • 2 have delivered presentations or papers at national conferences • 3 have won national or international awards for specific accomplishments and/or life work • 1 had a national award given to a women’s group mentored by the participant’s NGO Notes on Internet Search Tools While current Internet search tools are powerful, it is important to note that none of them, even the largest and most popular ones such as Google or Yahoo search the entire Internet. Most only search a small fraction of sites deemed to be of most interest to those who use their sites. In addition, the number of sites they “index” to appear as results for individuals using these tools is even smaller. Lastly, each search tool uses its own unique approach, often combining a mix of computer algorithms and human judgment, to capture, sort and prioritize results, which is why searches for the same term on different tools can yield different results. The tools selected for this project include: • AllAfrica.com English and French-language databases of African news (www.allafrica.com)—a searchable archive of 700,000 news stories from over 100 media organizations, including numerous African government-owned and private newspapers and news agencies, and UN agencies. • The Africa Center’s Contemporary Africa Database (http://africadatabase.org)—a Rockefeller Foundation-funded website containing brief biographical data on prominent Africans now living or who have died since 1950. Only a few participants appeared here, mostly government ministers. • Google (www.google.com)—is believed to be world’s largest public search engine with well over a one billion pages indexed. It ranks its findings based on the extent to which other pages refer or link to a page. It tends to lead people to results that already have been found and viewed by others, and is thus known in the industry as a “popularity engine.” It produced many potential links on participants which then had to be searched carefully for those that were relevant. • Dogpile.com (www.dogpile.com)—one of the world’s most thorough metasearch tools. Meta-tools do not create their own databases, but instead pull results from databases gathered by other search engines. Dogpile draws results from About, Ask Jeeves, FindWhat, Google, LookSmart, Overture, Teoma, and Yahoo. While this tool has a broad reach, it does not display the full set of entries indexed on sites it pulls from. Thus a search for “John Doe” on Dogpile will not yield as many hits from Google as a search on Google alone will produce. This was the most useful search engine tool for quickly determining if any information was available about the participant on the Internet. • AltaVista (www.altavista.com)—is one of the world’s largest search engines with over 400 million web pages indexed. It was useful for this project because unlike Google, it searches and indexes results from a number of international websites in French, Portuguese and other foreign languages. • Voila.fr (www.voila.fr)—is a French language website based in France, which produced hits on several francophone participants that were missed by the other search engines. ANNEX H ATLAS/AFGRAD IMPACT STUDY Proposed Methodology Memo submitted to: Cristin Springet, CTO By Andrew Gilboy Aguirre International Team Leader January 11, 2004 ______________________________________________________________________________ The methodology proposed below expands on the framework put forward in USAID's Terms of Reference and in the Technical Proposal submitted by Aguirre International to USAID. These are summarized in Part A below. Part B expands and modifies this methodology as it evolved with input from the ATLAS evaluation team. Part B therefore becomes the proposed methodology for this study. A. Comments on the Methodology Proposed in the TOR 1. The TOR restated the purpose of the "impact study" that was written in 1990 for the original ATLAS project paper : The purpose of the Impact Study is to establish a body of evidence that USAID-sponsored U.S. academic training has been critical to the development process and has made direct contributions to economic and social growth. The evidence should be expressed in verifiable and quantitatively specific terms including employment creation, investment earnings, productivity, mortality/morbidity, cost benefit ratios, rates of return, and multiplier effects. The study would not be concerned with the training process; it would deal directly with the quality and relevance of the training content, effects on the participant’s career, and the impact resulting from utilizing the training. Selected cases may also be chosen to document the unintended results of training. Institution building issues in universities, government ministries and other public sector institutions should be considered in selecting case studies.... The team debated the Impact Study "purpose" mandated above and drew the following conclusions: a. The first sentence, if taken literally, would prejudice the entire Impact Study, reducing it to documenting that the program did in fact contribute to economic and social growth in Africa. Instead, the team, with USAID support, opted to conduct an objective Impact Study that relied on a reliable methodology to ascertain whether there were results that might have occurred linked to USAID's investment. This methodology would include some new elements to enhance the credibility of the findings. b. The team agreed to design a study that was data-driven and relied as much as possible on quantitative evidence of impact, as suggested in the second sentence in the paragraph above. c. The team agreed that the study would not appraise the management of the program or the contractor's performance. d. The team agreed to consider the effects of training on participant careers. e. The team proposed to go beyond individuals, as implied in the "indicators" listed below, to discover whether impact occurred at higher levels, such as the organization, institution, sector, nation or region. f. The team accepted the mandate to seek out unusual examples of noteworthy or unintended impact suggested above. g. The team accepted the idea to include institutions in its population to be surveyed. 2. The original ATLAS project log frame included the following "indicators" against which it was suggested that a subsequent Impact Study could assess progress towards achievement of the ATLAS goals and objectives. The "indicators" listed were: • Strengthened programs in educational and training institutions, particularly in scientific, technical and economic fields; • Research institutions expand and improve their human capacities to carry out research relevant to African development, particularly for increasing agricultural productivity and technologies; • Public sector institutions show improved equity and efficiency in providing key services (health, education, transportation, etc.); • Increased indigenous capacity among African countries to manage their economies; • Increased human capacity to support the development of the private sector in African countries; and • Increased capacity among women to fill leadership and non-traditional roles. 3. The "purpose" stated in the ATLAS project paper was "to strengthen leadership and technical abilities and enhance professional performance of individuals serving in African public and private sector entities, including universities, research centers, and other key development institutions." 4. The "verifiable" indicators listed were: a. For ATLAS graduates to perform well and make significant contributions to key African development institutions, in the following areas: • Employment of the individual in key African development-related institutions (educational and training institutions, research institutions, public sector agencies, financial sector institutions) or in productive private enterprise; • Level of authority and responsibility and promotion record of the individual; • Important personal accomplishments on the job (e.g., technology generation, policy analysis or implementation, management innovations); • Immediate impacts of the individual’s actions on organizational decisions (e.g., policies, resource allocations, strategies, management systems and processes); and • Authority and influence of the individual as perceived by knowledgeable others. b. The performance of female graduates, as measured by the above indicators, matches that of the male graduates." 5. The TOR provides information about the ATLAS log frame only. Its predecessor program, AFGRAD, is included in the TOR but without any information about its indicators or program goals. 6. To measure most of the indicators indicated above, while respecting the limitations of the TOR, the team proposes an alternative methodology, which is described in Part B. 7. The TOR and Technical Proposal included a detailed task list that the team accepted as valid for its work. Part B: Modified Methodology 1. Framework. Kirkpatrick's levels of evaluation are put forward in the TOR and Technical Proposal as the structure to use in organizing information gathering and analysis. Because there appear to be national leaders and high ranking officials among the ATLAS/AGRAD alumni whose contributions may be felt beyond or outside the limits of an institution, the study will seek to capture this type of impact as well. The team agreed to modify the 4-level Kirkpatrick framework to add a fifth that would detect "higher-level impact," such as sectoral, national, regional or international. The Kirkpatrick levels that will therefore be used are: Level One: Reaction: the trainee's impression of the program; the level of satisfaction with the course, trainer, pace of instruction, content and materials; Level Two: Learning: the acquisition of skills and knowledge from the training; Level Three: Application: the performance of the trainee on the job following training; Level Four: Institutional Results: changes that the trainee's performance brought to the organization in efficiency, productivity, or profitability; Level Five: Higher-Level Results: changes that the trainee's performance brought directly to a nation, region or beyond to an international sector or institution. Change attributed to training at Levels 2, 3, 4 or 5 can be viewed as impact. The Impact Study will analyze impact at the two higher levels (Five and Four) in particular, and at Level Three, recognizing that data collected on levels 2 and 1 is important but less indicative of the impact USAID is seeking. This having been said, impact linked to the USAID intervention ("training") provided cannot occur without the acquisition of KSA (Knowledge, Skills and Attitudes) by the participant (Level 2). 2. Approach. The Aguirre Technical Proposal describes the traditional tools that would be used by the team to gather information, such as survey instrument, focus group and document review. That document should be consulted for additional information concerning the methodology that will be used. Building on this, the team then devised the following approach to gathering information that would help answer the questions raised in Part A regarding impact. The Kirkpatrick framework described above will be used wherever applicable, such as on the participant questionnaires, in focus group protocols and in presenting the study's findings. a. Participant-level Data Collection: The team reviewed the access to, and quality of, data kept by the contractor (Africa-American Institute) and determined that it might be possible to fashion a survey that could produce statistically-significant data with relatively low rates of error. If such a quantitative approach could be realistically implemented in the face of the many constraints faced, the team will be able to analyze findings and draw conclusions with a known, and high, level of confidence. Participants will provide information about perceived impact at multiple levels – individual, organizational, institutional, sectoral, national, etc. Because the answers offered by respondents will be grounded (that is, they will be considered valid only when backed up by examples accepted by the team), and triangulated, the team will be able to determine whether and what impact occurred. i. Survey Population. A detailed memo written by the Team Leader on December 15, 2003 details how the team arrived at a stratified, weighted survey of the 3,219 ATLAS/AFGRAD participants. ii. Country Origins. The memo tracks the decisions made by the team to select participants first by country, with no statistical impact on the standard deviation. The effect of this decision was to reduce the number of countries with participants that the team would have to locate from an estimated 42 to 27. iii. Gender Considerations. The team has included gender as required criteria in the sampling. The participant survey instrument will include questions into issues of gender and impact. iv. Administering the Questionnaire. After testing the draft questionnaire, the team will send the final version, in French, English or Portuguese, to the 400 to 500 names on the list generated by the team's data analyst. The questionnaires will go to all selected names wherever they may reside. Of that group, a large number will necessarily reside in the seven countries to be visited by virtue of the number of ATLAS/AFGRAD participants hailing from those countries. To aid in the distribution (and follow-up to ensure receipt) of the questionnaires, the team will seek support from AAI (for current addresses), the Missions (where appropriate) and via various African networks. We envisage getting temporary help from returned Peace Corps volunteers, African students in the U.S. on a temporary basis, and from others (former participants), to get the questionnaires out to, and back from, the selected participants. v. Entering Data from the Questionnaires. As the questionnaires are returned, the data entry will be organized in Africa by language: English will be entered in Ghana, French in Mali and Portuguese in Mozambique. This will limit errors due to hand-writing characteristics unfamiliar to Americans, lower cost and increase reliability of the data. vi. Data Collection, Management and Analysis. The entered data from each questionnaire will be sent to the team's data analyst electronically. Periodic reports will be distributed to the team of tables that will have previously been identified to show participant answers to key impact questions. b. Country-level Data Gathering. Seven countries were selected following a rigorous method described in the memo of December 15th. The countries are: Mali, Ghana, Benin, Uganda, Namibia, Mozambique and Madagascar. The purpose of the country visit is to gather supplementary information on impact via methods that reach beyond the limits of the individual questionnaire. It is at the country level that the team will focus on measuring higher-level impact and uncovering unusual examples of changes introduced by participants that may not be captured elsewhere. In each country the team (one or two team members, with support from local coordinators) will… i. conduct a focus group of returned participants ii. hold meetings with institutions where several or more participants have worked, iii. meet with any returned participant ("alumni") associations, and iv. interview and brief USAID personnel. c. University-level Data Gathering. The team leader will visit a representative sampling of U.S. "receiving" institutions where ATLAS/AFGRAD participants obtained their degrees. The purpose of these visits is to explore whether there are ongoing links between the institutions and participants, note any unusual or unintended impact and assess how the institution viewed the program. d. Other Data Gathering. The team will gather additional information on participant impact in the following way: i. Conduct an Internet search to discover whether participants produced research, books, articles, created associations, achieved recognition or did something that might have escaped notice through the two data-collection methods described above; ii. Follow-up on leads from participants interviewed of outstanding participants who may have escaped the team's notice; iii. Informally identify participants in the Washington and New York area who can be telephoned or interviewed for information on impact; iv. Interview former AAI program managers for their views on the impact of these two major USAID investments. 3. Innovations. Impact at multiple levels will be assessed using the approach described above. There are two innovations in the team's approach, as noted below: a. Statistically-significant participant data. Nearly all impact studies of participant training programs administer a questionnaire to returnees. What is unusual is the opportunity to introduce complex quantitative methods in the selection of those participants to be interviewed that can be implemented given the myriad constraints affecting a program operated in 43 countries across an area three times the size of the continental United States for over 35 years. The team is committed to trying to administer a questionnaire to this stratified random sampling in order to be able to analyze the respondents' answers and extrapolate findings to the entire ATLAS/AFGRAD program with a relatively high confidence level. b. Use of the Internet to find impact. This innovative idea emerged in discussions about ways to find out whether participants had published or produced something noteworthy, especially those who may not reside in the seven countries the team will visit. In those countries, a team member may hear of outstanding participant contributions from focus groups or during the week-long visit. In the other 19 countries, no such possibility can happen. The team hopes that through careful Internet searching, and by following-up on participant leads, that examples of impact at higher levels (if they exist) might be captured. The methodology described above will lead the team to gathering and analyzing data to respond to the principal questions that the Impact Study of the ATLAS/AFGRAD programs is charged with answering, as put forward in the TOR and Part A above. Although an ambitious methodology to implement with limited resources and time over a large area, the team accepts the challenge and anticipates being able to present findings about USAID's investments based on solid quantitative and qualitative evidence. ANNEX I EXAMPLES OF DISTINGUISHED AFGRAD/ATLAS ALUMNI ______________________________________________________________________________ The listing of Alumni on the following page was furnished by the Africa-America Institute to the Assessment Team. DISTINGUISHED AFGRAD/ATLAS ALUMNI Mr. Mamadou Dia, a native of Senegal, graduated from the Wharton School of Business under the AFGRAD program. Mr. Dia served as an economic advisor to the former President of Senegal before joining the World Bank, where he has held several key management positions. Currently, Mr. Dia occupies the post of Country Director for Guinea, Sierra Leone and Liberia. Mr. Dia is an international specialist in public sector management in Africa. In his book, “Africa’s Management in 1990s and Beyond,” he proposes an analytic framework which takes into account the influence of culture in resolving some of Africa’s management issues. His approach has been adopted and adapted with the Bank as a component of its overall agenda for enhancing institutional capacity in Africa. Dr. Isaac Wolde-Ab earned his Ph.D. in Chemistry at the University of Michigan as an AFGRAD fellow. As President of the University of Asmara in Eritrea, Dr. Wolde-Ab is trying to build and revitalize a university after the devastating effects of thirty years of war. Under his leadership, the university is playing a leading role in the process of nation building. The university is deliberate in developing a work force with the technological skills to usher in an export-oriented economy. The University of Asmara has developed the Dual Linkage Model, considered a model of partnerships, which joins the university to local public and private sectors, as well as to advanced institutions within the U.S., Europe and Australia. Dr. Johnson Jato participated in research of a previously unknown vine in the Cameroon rain forest which contains a chemical that blocks the reproduction of the AIDS virus. Concerned if the plant could be cultivated away from its usual habitat, Dr. Jato grew the plant in Yaounde. Leaves from these plants were tested by the National Cancer Institute and found to contain the active compound. He is presently working on possible production by tissue culture. Dr. Jato is the first Vice-Dean of the Faculty of Medicine and Biomedical Sciences at the University of Younde. As Vice-Dean, Dr. Jato is establishing scientific collaboration with institutions in other countries to further research and training in medicine and pharmacy. Dr. Jato received his Ph.D. in Pharmacy from the University of Wisconsin under the AFGRAD program. Ms. Setcheme Mongbo earned her Master’s degree in Sustainable International Development from Brandeis University under ATLAS. Her Master’s thesis, “Gender and Underdevelopment: The Impact of Mothers’ Condition on Child Welfare in Southern Benin” identified the contributions of mothers to children’s education, health care and food; explored factors which limit or encourage their contribution; and formulated a development program based upon her findings. After graduating, Ms. Mongbo returned to Benin and founded the Programme d’Appui aux Femmes pour le Development Durable (PAFeDD), a grassroots organization addressing gender disparities. Within its development efforts, PAFeDD has promoted the initiation of micro￾savings and credit groups to facilitate poor women’s access to financial resources, and promoted agricultural cooperatives to encourage female economic associations. Dr. Ablade Glover of Ghana is a scholar, administrator and noted painter who established a national gallery for contemporary Ghanaian artists which opened in 1993. He has held numerous exhibitions in Nigeria, Cote d’Ivoire, Zimbabwe, and Sierra Leone, as well as in Europe and the United States. Dr. Glover earned a Ph.D. in Education at Ohio State under AFGRAD and is currently Dean of the College of Art and Head of the Department of Art Education at the University of Science and Technology in Kumasi. Mr. Hyacinthe Toure, an AFGRAD alumnus from Cote d’Ivoire, earned a Master of Science in Computer Science at SUNY-Buffalo. Mr. Toure’s founded a consulting firm for computer services in Abidjan which helps public and private organizations make decisions about electronic communications to improve organizational and human performance. In addition, his firm conducts seminars and workshops on communication technologies which help customers such as universities, laboratories and businesses to better share information. Dr. Wellington Otieno received a Ph.D. in Entomology as an AFGRAD fellow from the University of California at Berkley. Upon returning to Nairobi, Dr. Otieneo joined the International Center of Insect Physiology and Econology (ICIEP) to conduct research on the biological control of mosquitoes. Dr. Otieno helped to develop an integrated pest control system that uses insects’ natural enemies rather than chemical insecticides. The methods were easily put into place by farmers as they are socio-culturally acceptable, as well as environmentally safe and economically attainable. Dr. Otieno was involved in establishing an African Regional Post￾Graduate Program and is involved in ICIPE’s launch of a graduate school in Kenya for insect science. Ms. Christiana Morgan, an AFGRAD alumnus from Guinea, earned a M.S. in Mineral Economics at Michigan Technological University. She is presently Chief of the Promotion Section of the Center for Promotion of Mining Development at the National Direction of Mines in Conakry. Her most recent publication is “The Role of the Mining Industry in Guinea’s Development.” Dr. Chris Bakwesegha of Uganda earned both an M.A. and Ph.D. in Town Planning from Rutgers University under AFGRAD. Dr. Bakweshega’s interests lie with human rights and the promotion of multi-party democracy in Africa. Dr. Bakwesegha headed the Division of Conflict Management within the Organization of African Unity to the UN. Presently, Dr. Bakwesegha is Deputy Permanent Observer to the Permanent Observer Mission of the OAU. ANNEX J SCOPE OF WORK AND MODIFICATIONS ______________________________________________________________________________ (Note: This Annex contains the Scope of Work written in October, 2003 for this assessment. A summary of modifications made by the team is appended to the last page.) STATEMENT OF WORK IMPACT STUDY OF USAID’S AFRICAN TRAINING FOR LEADERSHIP AND ADVANCED SKILLS (ATLAS) PROJECT I. Background The African Training for Leadership and Advanced Skills II Project (ATLAS II) is the fifth in a series of projects designed to address Africa’s lack of trained human capital in order to create an enabling environment for sustainable growth. The ATLAS Project began in 1990. Like its predecessor, the African Graduate Fellowship (AFGRAD) Project, which began in 1963, a prominent feature of ATLAS is the cooperation among U.S. universities, USAID, and African countries to address sub-Sahara Africa’s needs for advanced academic training. The mainstay of this cooperation is the tuition relief granted by U.S. universities on the basis of academic promise and leadership potential. The ATLAS Project was authorized on July 23, 1990 by USAID’s Africa Bureau. The goal of the project is to improve the performance of African institutions and organizations to plan and promote sustainable development in Africa. Its purpose is to strengthen leadership and technical abilities and enhance professional performance of individuals serving in African public and private sector entities, universities, research centers and other key development institutions. The project has had two main components: 1) the participant training scholarship program with its attendant activities for selection, placement and management of African students in degree and postgraduate non-degree programs at U.S. universities; and 2) the post-training program which promotes professional competence among graduates of U.S. funded training and reinforces their ability to contribute to African development. On September 25, 1995, the Africa-America Institute (AAI) was awarded the contract for the second five-year segment of the implementation of the ATLAS Project. On September 28, 2000 an amendment to the contract provided a no-cost extension to September 28, 2003. As of September, 2002 the USAID contribution for training and administrative costs provided 84% of total costs. Funds for training were provided through USAID mission OYB transfers (69%) and have been supplemented by core funds for administrative costs and professional enhancement activities (15%). U.S. universities contributed 16% of total program costs through scholarships and tuition relief. II. Goals and Objectives of the Study The ATLAS Project Paper, written in 1990, calls for a study to assess the impact of ATLAS and AFGRAD training on the attainment of Development Fund for Africa (DFA) targets and objectives. “The purpose of the Impact Study is to establish a body of evidence that USAID￾sponsored U.S. academic training has been critical to the development process and has made direct contributions to economic and social growth. The evidence should be expressed in verifiable and quantitatively specific terms including employment creation, investment earnings, productivity, mortality/morbidity, cost benefit ratios, rates of return and multiplier effects. The study would not be concerned with the training process; it would deal directly with the quality and relevance of the training content, effects on the participant’s career, and the impact resulting from utilizing the training. Selected cases may also be chosen to document the unintended results of training. Institution building issues in universities, government ministries and other public sector institutions should be considered in selecting case studies...” While the original intent of looking at investment earnings and cost benefit ratios is beyond the scope and resources available for this evaluation, certain program elements (such as selection based on leadership qualities) will be reviewed for their impact on the individual and development activities. Under this Task Order, the evaluation team will assess how well the ATLAS Project program indicators met its purposes and goals. In the original logical framework of the 1990 project paper, six goal-level indicators were established. By aggregating the experience of several missions in each of the six areas, the ATLAS impact evaluation will draw conclusions about the contribution of long-term U.S. participant training in achieving USAID’s broader development objectives. The established indicators are: • Strengthened programs in educational and training institutions, particularly in scientific, technical and economic fields; • Research institutions expand and improve their human capacities to carry out research relevant to African development, particularly for increasing agricultural productivity and technologies; • Public sector institutions show improved equity and efficiency in providing key services (health, education, transportation, etc.); • Increased indigenous capacity among African countries to manage their economies; • Increased human capacity to support the development of the private sector in African countries; and • Increased capacity among women to fill leadership and non-traditional roles. In order to meet these goals, the purpose in the logical framework was “to strengthen leadership and technical abilities and enhance professional performance of individuals serving in African public and private sector entities, including universities, research centers, and other key development institutions.” The verifiable indicators for this purpose are: A. For ATLAS graduates to perform well and make significant contributions to key African development institutions, in the following areas: • Employment of the individual in key African development-related institutions (educational and training institutions, research institutions, public sector agencies, financial sector institutions) or in productive private enterprise; • Level of authority and responsibility and promotion record of the individual; • Important personal accomplishments on the job (e.g., technology generation, policy analysis or implementation, management innovations); • Immediate impacts of the individual’s actions on organizational decisions (e.g., policies, resource allocations, strategies, management systems and processes); and • Authority and influence of the individual as perceived by knowledgeable others. B. The performance of female graduates, as measured by the above indicators, matches that of the male graduates. Aguirre International Understandings It is our understanding that this evaluation is focused on the results and impact of the ATLAS/AFGRAD program on alumni and institutions and not on the management of the programs and activities. It should be noted that this impact assessment is not “scientific proof” of professional or institutional change, but will present reasonable findings that the ATLAS training and education contributed to positive change, …or not This report does not provide answers to what education or training programs or types USAID should select because such questions must be considered in relation to a Strategic Objective team’s strategy, the capacity of its partner institutions and the results targeted. This proposal and proposed budget represent the best estimate to conduct all the activities proposed in the Statement of Work for this effort; however, as stated in the proposal, the proposed budget and timeline are contingent on certain assumptions that were necessary in order to meet the funding constraints. These assumptions include (a) that the collection and quality of contact information and participant data must be of good quality, (b) the proposed timeline will move smoothly, and (c) that the final destinations, and proposed combined trips to multiple destinations in a single trip, are possible within the estimated travel costs. III. STATEMENT OF WORK TASKS This is a final evaluation to assess the impact of ATLAS and AFGRAD training in achieving development objectives at the country and the regional level by providing long-term academic training to individuals with leadership potential. The major focus of this study will be to examine the link between long-term academic training and participation in follow-on professional activities with the success of the individual and the impact on organizations, countries, and the region. The specific tasks for this impact study follow: Task 1. Develop the approach and methodology and submit to USAID/W for review. This will include establishing performance indicators (those mentioned in the project paper as well as others the evaluator may develop), instruments for interviewing and surveying participants, stakeholders, and other partners. It will focus on strategies for arriving at quantifiable and verifiable evidence of the impact returned participants make on the institutions and sectors in which they are employed. Task 2. Review select program elements and the impact they have had to determine the individual success of the participant and the impact on the institution/organization and nationally. Examples of program elements to be considered are selection criteria and method of selection. Task 3. Review professional enhancement activities undertaken in Africa by the core contractor. This will include interviews with alumni associations and members of African professional societies to determine if a sense of cohesiveness has developed through the AFGRAD/ATLAS activity; which alumni groups have had an impact nationally and regionally; and how this has been sustainable. Features of successful associations will be noted. Task 4. Select samples based on participant interviews and questionnaires regarding the training programs and post training employment taken at intervals of 5 years, and demonstrate what impact the long-term training had on the individual’s career development. The sampling should be taken from various stages of the AFGRAD/ATLAS program, going back to the 1970s where possible. Task 5. Collect samples of host country governments’ and sponsoring institutions’ comments supported by data if available, on the impact that the AFGRAD/ATLAS program has had on organizational and national development through their organization’s or country’s participation. Task 6. Collect samples based on interviews and questionnaires of those participants who have been forced to leave their home countries through civil strife or natural disaster to determine how the training affected the professional mobility of the individual, host country development, and ultimately development in the region. Task 7. Collect data showing to what extent linkages with U.S. universities have continued and what impact this has had on the participant’s professional development. IV. PROPOSED TASK PLAN OUTLINE AND TIMELINE The ATLAS Impact Evaluation study will be developed and implemented in accordance with this proposal as guided and approved by the USAID/W CTO. The timeframe for the study is September 30, 2003 to April 30, 2004. The following table represents an illustrative project schedule and timeline. In order to be responsive to Mission needs, windows of opportunity for meetings and interviews, and other unforeseeable events, the staging and sequencing of the activities in the project schedule will be as flexible as possible, while still adhering to the proposed end date. Table 1. Proposed Timeline/Project Plan for a 6-Country, 4-University Study of ATLAS/AFGRAD ILLUSTRATIVE TIMELINE AND PROJECT PLAN Phases/ Dates No. Tasks Phase One: Information Gathering and Planning 1.1 Document Review 1.2 Team-Building Meeting with USAID Meeting with Africa-American Institute Project Management 1.3 Preparation of Methodological Strategies, Impact Study Schedule, and Work Plan 1.4 Selection of Countries and Universities to be Visited 1.5 Review of all Participant Contact Information 1.6 Drafting of Survey Instruments for Participants 1.7 Testing of Survey Instruments 1.8 Development of Interview Protocols for Participants 1.9 Development of Protocols for Host Country Organizations and Stakeholders 1.10 Selection of Sample Population to be Surveyed and Interviewed 1.11 Initiation of data collection prior to field research 1.12 Preparation of Travel Schedules for African and US Field Visits: Countries and Universities 1.13 Arrangement of In-country Appointments 9/30/03 through 12/30/03 Deliverables: Methodological Strategy, Impact Study Schedule, Work Plan, Survey Instruments, Interview Protocols, Travel Schedule Phase Two: Data Collection and Field Visits* 2.1 Team Leader and Participant Training Specialist Travel to Senegal Data Collection in Senegal Team Leader and Participant Training Specialist Travel to Benin Data Collection in Benin Team Leader Travels to Ghana Data Collection in Ghana Team Leader and Economic Development Specialist Travel to Uganda Data Collection in Uganda Team Leader and Economic Development Specialist Travel to Mozambique Data Collection in Mozambique Team Leader returns to US Participant Training Specialist and Economic Development Specialist Travel to Ethiopia Data Collection in Ethiopia Participant Training Specialist and Economic Development Specialist return to their respective countries 2.2 Participant Training Specialist Travels to Regional ATLAS Alumni Conference in Africa 1/15/04 through 3/15/04 2.3 Team Leader Travels to ATLAS Conference in New York 2.4 Team Leader Travels to ATLAS Partner Universities in Mississippi, Wisconsin, Texas, and Washington State Deliverables: Trip Reports * Field Visits will include meetings with USAID officials, core contractor representatives, host government officials, other stakeholders and partners, alumni associations, and participants. The evaluation team will hold initial briefings and debriefings with the USAID Mission in each country. Phase Three: Data Tabulation and Analysis 3.1 Clean, Compile, Analyze Quantitative Data 3.2 Compilation of Interview Responses 3.3 Drafting of Report Chapters 3.4 Team Meeting to Discuss Findings and Recommendations 3/16/04 through 4/10/04 Deliverable: Interim Report on Findings/Recommendations Phase Four: Presentation of Findings and Recommendations 4.1 Preparation of Summary of Findings (40 pp. with 2 page Executive Summary) 4.2 Preparation of Report (PowerPoint) Presentation for USAID/W 4.3 Meeting to Present Findings and Recommendations and receive comments 4.4 Preparation of Draft Report (40pp with 2p Executive Summary) 4/11/04 through 4/22/04 Deliverables: Power Point Presentation to USAID, Draft Report with Executive Summary Phase Five: Completion of Final Draft and Delivery 4/23/04 5.1 Incorporate USAID Comments and Corrections into Final Report through 4/30/04 5.2 Production of Final Report (40pp with 2p Executive Summary) Deliverable: FINAL REPORT, bound hard copy and in electronic form V. TECHNICAL APPROACH TO THE EVALUATION The impact study will be conducted in five phases and the technical approach follows the outline of the preceding table, “Proposed Timeline/Project Plan for a 6-Country, 4-University Study of ATLAS/AFGRAD”. Phase One: Information Gathering and Planning It will be necessary to carefully structure and plan the various steps leading up to and including the data collection—especially survey and interview protocol development, database development, and communications. Document Review. To conduct the Document Review, the ATLAS/AFGRAD evaluation team will require background materials and the relevant available documentation on the status of the ATLAS Project in order to get an historical perspective and a thorough understanding of the evolving nature of this project over time. It is anticipated that the USAID CTO will provide the evaluation team all of the necessary specific background materials on the project, such as previous evaluations, technical reports, annual reports, participant databases, etc., in order to prepare and conduct an informed evaluation study. 1.2 Team Building Meeting. Following their review of documentation, evaluation team members will travel to Washington, DC, for a team-building meeting with the USAID CTO, relevant USAID officials, and a representative of the ATLAS contractor staff. The purpose of the meeting is to further deepen the understanding of the team members about the goals and objectives of the program, the data and information needs of USAID, and enlist the support of all the interested parties in this participatory evaluation study. 1.3 Meeting with Africa-America Institute Project Management. Following the Team Building Meeting, two members of the evaluation team will visit the Africa-America Institute in New York for briefings on the current status of the project in each country, review and receive electronic copies of participant databases, and acquire contact information for in-country stakeholders, the Executive Committee of Graduate Deans, and participating ATLAS universities. 1.4 Preparation of Methodological Strategies, Impact Study Schedule, and Work Plan. The team as a group, directed by the Team Leader, will develop three documents: a more detailed and targeted Methodological Strategy than is proposed here; a fleshed-out Impact Study Schedule, and a Work Plan that shows the responsibilities, travel, and deliverables of the evaluation team members. 1.5 Selection of Countries and Universities to be Visited. Following initial meetings with USAID and document review, a country selection will be proposed by the evaluation team based on criteria they identify as key to turning out an objective evaluation. The recommended countries will be reviewed and approved by the CTO. Countries listed in this Statement of Work (Senegal, Benin, Ghana, Uganda, Mozambique, and Ethiopia) are illustrative and are included for budget-development purposes. These countries, however, meet certain criteria of language, geographic region, and travel/per diem cost. 1.6 Review of Participant Contact Information. The contractor will arrange for appointments and notification of participants be done “up front” ahead of international travel when possible. The evaluation team will take advantage of opportunities to survey participants concentrated in certain locations in the U.S. (World Bank) and alumni gatherings at other sites. 1.7 Drafting of Survey Instruments for Participants. A survey is one among many instruments that evaluators can use to help understand the bridge between training (the transfer of KSA) and change or performance improvement. The key question the assessment seeks to answer is whether development impact resulted from investments in training. Kirkpatrick’s “Level Four” impact requires multiple assessment approaches to be able to measure organizational performance changes. The survey is but one instrument, albeit a crucial one in light of the few alternatives at the disposal of evaluators in Africa. The evaluation team will develop quantitative survey instruments to interview ATLAS/AFGRAD alumni about the accomplishments, career changes, and institutional impact of the program. The protocols will derive from document review, USAID/W management and reporting needs, and from recommendations by the institutional contractor and the Executive Committee of Graduate Deans. Draft protocols will be submitted for review by the USAID/W CTO. 1.8 Testing of Survey Instruments and Modifications. Draft survey instruments will be pilot tested with 10 ATLAS/AFGRAD alumni in two of the countries selected for site visits. Following the piloting, appropriate modifications will be made to the instruments and shared with the USAID/CTO prior to the launch of the field surveys. Where feasible, the contractor will use Internet technology to disseminate surveys. To the extent allowable by time and resources, most surveys will consist of face-to-face interviews. 1.9 Development of In-Depth Interview Protocols for Participants. After conducting the document review and receiving briefings by USAID/W staff and Africa-America Institute staff, the evaluation team will draw up an interview protocol for in-depth interviews of program alumni and submit it to the USAID/W CTO for approval. 1.10Development of Protocols for Host Country Organizations to be Surveyed and Interviewed. The evaluation team will develop an interview protocol or protocols appropriate for Mission staff, host government officials, core contractor representatives, alumni association officers, and other stakeholders that may be identified by the USAID CTO or through the document review. CTO approval will be obtained for these protocols. 1.11Data Collection Prior to Field Research. In the team-building meeting, the evaluation team will work with the CTO to design and structure innovative ways to begin data collection prior to the team’s departure to do field research. 1.12Selection of Sample Population to be Surveyed and Interviewed. The Team Leader, working with the Data/Statistical Analyst, will draw data samples from participant alumni databases and other databases that will have the most current contact information. It is understood that there will be no comparison groups used for this evaluation study. A survey sample size will be suggested by the evaluation team, following document review, country selection, and review of alumni database information. 1.13Preparation of Travel Schedules for Africa and U.S. Field Visits: Countries and U.S. Universities. The Team Leader and traveling members of the team will coordinate their international travel schedules with their respective in-country support specialist to avoid holidays or other special events in the countries to be visited. The Team Leader will be responsible for scheduling his travel to and appointments with U.S. universities to be visited. All travel schedules will be furnished in advance to the USAID/W CTO and country clearance obtained prior to international travel. 1.14Arrangement of In-country Appointments. It will be necessary to select and hire an in￾country support specialist in each country for approximately four days in order for him/her to find office space and focus group space, hire a local administrative assistant, locate participants, schedule appointments with stakeholders, screen for focus group candidates, and perform other duties as determined by the Team Leader. The in-country support specialist would facilitate the appointments and travel of the U.S. evaluation team members while on-site in the countries. In-country appointments will include entrance and exit debriefings with the USAID Mission staff, meetings with relevant host government officials, meetings and appointments with alumni and alumni associations, meetings with professional groups, and appointments with other stakeholders. Deliverables: Methodological Strategy, Impact Study Schedule, Work Plan, Survey Instruments, Interview Protocols, Travel Schedule Phase Two: Data Collection and Field Visits Information-sharing and data collection will be accomplished through field visits to six countries. It is anticipated that these visits will take place between January and March 2004. At a minimum, five to six days in-country, plus travel time, will be needed to carry out the following activities in each country, listed below. Travel and level of effort are also included for site visits to four (4) universities (8 days), travel to and from the Washington, DC and AAI in New York (4 days), trips to Washington, DC at the beginning and end of the study (3 days), and a trip to one Regional ATLAS Alumni Conference in Africa (4 days). Proposed In-Country Activities • Briefing of key Mission staff; • Briefing by in-country core contractor representatives, where possible; • Reviewing/training the Education/Logistical Specialist on protocols; • Meeting with appropriate Ministry and host government staff; • Meeting with representative of local ATLAS/AFGRAD alumni officer(s); • Focus Group with Alumni; • Survey and Interview of Alumni • Meeting with representative of local professional association(s) • Interview of Alumni employers • Debriefing of Mission staff In order to maximize the labor and maintain evaluation team integrity, the following international travel schedule is proposed (see Table 2). Two persons from the evaluation team will travel together through four of the six illustrative countries. In the case of Ghana and Uganda, the Team Leader will conduct the data collection without an evaluation team partner. Table 2. Proposed Deployment of Evaluation Team Members in the Field IILLUSTRATIVE DEPLOYMENT OF EVALUATION TEAM MEMBERS Country Visits Team Leader/Evaluation Specialist Participant Training/ Evaluation Specialist Economic Development Specialist Senegal 9 9 Benin 9 9 Ghana 9 Uganda 9 Mozambique 9 9 Ethiopia 9 9 Note: The Data/Statistical Analyst will not take part in International Travel. In collaboration with the CTO, the contractor will explore the most productive, effective, and economical strategy for fielding the research team and to develop criteria for the final selection of countries. 2.1 Proposed Country Visits (Illustrative) • Data Collection Site Visit to Senegal. On this first visit the Team Leader will be accompanied by the Participant Training Specialist. • Data Collection Site Visit to Benin. The Evaluation Team for Benin will consist of the Team Leader and the Participant Training Specialist. Following the data collection activities, the Participant Training Specialist will return to his home in Mauritania. • Data Collection Site Visit to Ghana. The Team Leader will conduct the evaluation visit to Ghana without other team members. • Data Collection in Uganda. The Team Leader will conduct data collection in Uganda without other team members. • Data Collection in Mozambique. The Team Leader and the Economic Development Specialist will conduct the data collection in Mozambique. Following this task, the Team Leader will return to the U.S. • Data Collection in Ethiopia. The Participant Training Specialist and the Economic Development Specialist will meet in Addis Ababa to conduct the Ethiopia data collection. Following this effort, each will return to his respective countries. 2.2 Participant Training/Evaluation Specialist Travels to Regional Atlas Conference in Africa. If a Regional Atlas Conference takes place in a country scheduled for an evaluation team visit, the evaluation team member covering that country will attend the conference. However, if conference dates and travel schedules do not permit this, the Participant Training Specialist (from Mauritania) will attend the conference to speak with alumni and other stakeholders who are in attendance. 2.3 Team Leader and Economic Development Specialist Travel to AAI in New York. The Team Leader and the Economic Development Specialist travel to New York to interview AAI staff and former ATLAS and AFGRAD staff (such as Heather Monroe and Niameny M____). He will take this opportunity to interview Africa-America Institute staff about the program as well as alumni and other stakeholders who may be available. 2.4 Team Leader Travels to ATLAS Partner Universities in Mississippi, Wisconsin, Texas, and Washington State. The Team Leader will visit four universities that have hosted ATLAS/AFGRAD participants to interview faculty, project directors, administrators, and international staff connected with the program. The following list of states given is for illustrative purposes: Mississippi, Texas, Wisconsin, and Washington State, and will include at least one HBCU university. The final list will be determined in discussions with the USAID CTO. 2.5 Data Compilation. During this period of time, as data are gathered through site visits and in-country visits, they will be fed back to the Data/Statistical Analyst for screening and data entry. Deliverables: Trip Reports from the countries visited. Phase Three: Data Tabulation and Analysis 3.1 Clean, Compile, and Analyze Quantitative Data. The Data/Statistics Analyst will receive survey data from the field, organize it into appropriate databases, clean the data, and produce data runs for the evaluation team as needed. 3.2 Compilation of Interview Responses into Categories. The Team Leader will direct the evaluation team members to organize and analyze the survey open-ended questions, participant interview responses, and information from the various interview protocols and divide them into various headings for the report. 3.3 Drafting of Report Chapters. During this phase, each Evaluation Team member will be responsible for compiling and writing sections of the reports as assigned by the Team Leader. The chapters, with analysis, findings, and recommendations, will be submitted to the Team Leader for review and approval. Where deemed necessary, the Team Leader will request changes and modifications by the authors of the chapters. 3.4 Team Meeting to Discuss Findings and Recommendations. The contractor proposes that several virtual team meetings be held to discuss the findings and recommendations for the report. The Team Leader will organize and direct these meetings as needed. Deliverable: Draft Report on Findings and Recommendations Phase Four: Presentation of Findings and Recommendations 4.1 Preparation of Summary Findings. The Team Leader will direct the evaluation team in the development of a summary of findings and present it to the USAID/CTO one day prior to presentation. 4.2 Preparation of Report Presentation to USAID. The Team Leader will arrange for a presentation of the findings to USAID/W staff in a format agreed upon with the USAID CTO. 4.3 Meeting at USAID to Present Findings and Recommendations. The Team Leader and one other team member will attend a meeting at USAID/W to present findings and recommendations to the interested parties. Comments and input to the report will be received at that time for incorporation into the final report. Deliverables: Draft Summary, Power Point Presentation, Draft Report with Executive Summary Phase Five: Completion of Final Draft and Delivery 5.1 Incorporation of USAID Comments and Corrections into Final Report. The Team Leader will consider all comments received from the CTO and other stakeholders and incorporate these, as appropriate, into the final draft of the report, which he will submit to the contractor for production. 5.2 Production of Final Report. Five bound copies will be submitted to the USAID/W CTO by April 30, 2004. Electronic copies of the report, appendices, data sets, and periodic reports will be submitted to the USAID/CTO at then end of the evaluation study. 5.3 Deliverable: Final Report VI. METHODOLOGICAL APPROACHES TO THE EVALUATION Theoretical Basis for the Methodology The team for this assessment will select an impact evaluation methodology, originated by Donald L. Kirkpatrick and widely used throughout North America for over 40 years. The Kirkpatrick four-level model was adapted by USAID in its Best Practices Guides as a useful tool to assess impact. Briefly stated, it traces impact from training at four levels: • Reaction: the trainee's impression of the program; the level of satisfaction with the course, trainer, pace of instruction, content and materials; • Learning: the acquisition of skills and knowledge from the training; • Application: the performance of the trainee on the job following training; and • Results: changes that the trainee's performance brought to the organization in efficiency, productivity, or profitability. Change attributed to training at Levels 2, 3, or 4 is impact. The assessment is most concerned by impact first at Level Four, then at Level Three, although data collected on levels 1 and 2 is important. The contractor will also endeavor to trace the impact of the ATLAS Program from training to a “fifth level”—beyond changes to organizations—to the impact the program had on regions and/or countries. There are national leaders and high ranking officials among the ATLAS alumni who have made a difference in their sphere of influence and the evaluation will seek to capture this higher potential of ATLAS impact. The methods proposed in this assessment include interviews with key persons in USAID, a review of documents, a written survey administered to a weighted random sampling of former participants, individual participant interviews, a focus group, and interviews with other stakeholders. Following the administering of the survey instrument in person, team members will conduct a brief open-ended interview during which time valuable anecdotal information can be gathered and later shared during team meetings. Methodological Discussion The methodologies to be used in implementing this evaluation activity will be consultative and participatory, with all stakeholders represented, and guided by the clear need for information about past performance, current status, program impact, and future needs. Following the document review process, initial discussions with USAID/W, and a review of available contact and information databases, the evaluation team will submit a detailed plan of the methodology it proposes for the activity, bearing in mind the indicators and the needs noted in the SOW. The detailed plan of the methodology to be employed, draft instruments developed for information/data collection, the approach to analysis, and the reporting format will be submitted to USAID/W after initial meetings on data needs, understanding of technological capacities of participants, and the availability of current contact information for a targeted survey. The contractor will employ a traditional evaluation model, using both quantitative and qualitative methods of data collection and will gather information on and insight into the various components of the project through a variety of methods including the following: • Document Review. The primary means of gathering information and insight into the various components will be through the review of documents, formative evaluations, and reports, and interviews with participants and stakeholders in the program. • Focus Groups, Discussion Groups, and/or In-depth Interviews. This will include interviews and focus groups or discussion groups with USAID staff, Ministry personnel in relevant units, contractor staff, participants, alumni, professional association members, community members, and employers where available. • Specialization. Each of the experts on the team will focus on the area of his or her specialty, and the Team Leader will assign responsibilities and coordinate their activities to produce the deliverables. • Data Collection. The team will adopt the procedures for data gathering that will yield the best results, depending on the target audience, such as individual interviews, on-line survey, focus groups, targeted discussion groups, key informant interviews, open-ended interviews, unstructured interviews, etc. Relationship of Quantitative and Qualitative Methodologies The importance of quantitative data and its relationship with qualitative data should be recognized in this assessment. Both are useful in conveying the findings and explaining the conclusions. Individual face-to-face interviews will be conducted using a protocol containing questions that will obtain both quantitative data and qualitative information. The contractor will construct interviews and surveys in such a way that certain queries are presented to all interviewees. On-line surveys or other methods of surveying will be constructed primarily of quantitative items. The qualitative data obtained through discussion and focus groups, and informant interviews will enrich the quantitative data by providing context, description, elaboration, and effect. Focus/Discussion Groups In each of the countries to be visited, the design team will conduct at least one focus group with project alumni. The purpose of the focus groups, or group discussions, is to determine what tangible outcomes they perceive came from their participation in program, what activities they would like to see in the future programs, and what suggestions they may have for possible improvements to the project. The Team Leader and the evaluation team will work closely with the CTO to seek creative approaches to data collection and instrument development. We all must not lose sight of the fact that training assessments seek plausible findings—not scientific proof—about the nature of changes that may have occurred as a result of education and training programs funded by USAID. VII. Team Composition The ATLAS Project evaluation will be conducted over a seven-month period in six countries by a four-person multi-disciplinary team, with backup support in the U.S. and logistical support in￾country. The evaluation team will be comprised of: a Team Leader/Evaluation Specialist, a Participant Training/Evaluation Specialist, an Economic Development Specialist, and a Data/Statistical Analyst. Each team member should possess an advanced degree (Masters or above). None may have prior long-term professional association with the AFGRAD/ATLAS contractor or sub￾contractors; one or more must speak French and/or Portuguese fluently. Strong writing and word processing skills in English are a requirement; all must be familiar with computer word processing; all must be able to withstand a rigorous travel schedule to several African countries with minimal local support. The qualifications of the team members are detailed below: Team Leader The Team Leader is responsible for coordinating and directing the overall impact study effort; including editing of the entire report for consistency; preparation and submission of the draft and final study to USAID/W; liaison with USAID and contractor. Required: Five years’ experience working in a developing country, preferably in Africa, on human resources development activities; knowledge of USAID internal training management issues; experience serving as Team Leader on at least one large project evaluation for USAID or an international donor agency; understanding of evaluation methodologies and instruments; ability to organize data, manage a team, write and edit a complex report in English; graduate degree in social sciences, organizational development or economics or equivalent work experience. Fluency in French. Participant Training/Evaluation Specialist Required: Five years’ experience working on participant training projects; in-depth knowledge of American and African university systems; experience working in a developing country, preferably Africa on human resource or participant training activities; familiarity with USAID regulations and procedures in implementing training or managing development activities; experience in designing and evaluating participant training components or projects, preferably in a USAID context; experience serving on at least one large project evaluation for USAID or an international donor agency; graduate degree in social sciences, management, organizational development or equivalent work experience. Economic Development Specialist Required: Graduate degree and work experience in economics or business with emphasis on development economics or institutional economics; experience applying economic analysis in program and project design or evaluations in developing countries, preferably Africa. Experience in serving on evaluations or impact studies. Knowledge of economic development institutions and organizations in Africa. Data /Statistical Analyst Required: Graduate degree in economics or statistics. Experience working on development projects, preferably in Africa; knowledge of project evaluation methodology and design issues (outputs, measurable indicators, etc); ability to analyze complex data and reports. Logistical Support Logistics: The team will be responsible for hiring one local person in each country where necessary to provide logistical support. The experience and qualifications of the Education/Logistical Specialist for each of the six country visited will be an important factor in maximizing the time evaluation team members when they are in-country. The persons to be hired should have an understanding of the education system and education programs in their country, be of sufficient stature to make appointments with ministry and university officials, should have excellent communication skills, should be easily contacted and communicated with, and have some experience in information gathering and experience in setting up and conducting survey work. U.S. Backstopping Contract Manager, will work with the Team Leader and team members from the U.S. in preparing for the work in Africa, will support the team substantively during the evaluation and strategy development, and will be responsible for the finalization of deliverables produced by the team. Evaluation Specialist, will provide team coordination from the U.S. and provide feedback and comment on draft deliverables and final reports. Evaluation Specialists, also from the U.S., will revise, format, and produce the final report. Data Specialist. From the U.S. will assist team members with the data entry and manipulation, the production of data tables, and the initial analysis of quantitative data. Ethical Considerations The evaluation will be conducted by generally accepted ethical evaluation behavior. All members of the evaluation team will be expected to give 100 percent of their work time to this evaluation while employed under this task order. If an individual team member is unable to be available for a period of time, this will be discussed and negotiated by the contractor with the USAID CTO and interim plans will be made. VIII. REPORTS AND DELIVERABLES The evaluation team will deliver timely, organized, clear, and plainly readable deliverables for a variety of audiences. A. The following deliverables will be submitted to the USAID/W ATLAS/AFGRAD Coordinator as listed below. 1. End of Phase One: • Methodological Strategy • Impact Study Schedule • Report on Document Review Findings • Survey Instruments • Interview Protocols • Travel Schedule 2. End of Phase Two: • Trip Reports 3. End of Phase Three: • Interim Report on Findings 4. End of Phase Four: • Preliminary Draft Report • Presentation to USAID/W • Revised Draft Report 5. End of Phase Five: • Final Report B. The Final Report will incorporate the following illustrative outline. 1. Executive Summary 2. Background and Overview of the Project 3. Findings: by country; by region 4. Conclusions 5. Recommendations 6. Unresolved Issues 7. Lessons Learned/Best Practices 8. Persons Consulted 9. Survey Instruments 10. Interview Protocol Instruments C. Five (5) bound copies of the Final Report will be submitted to USAID/W along with report files and data files in electronic format. IX. SCHEDULE The evaluation team will conduct its data collection activities in Africa and the U.S. between September 30, 2003, and April 30, 2004. In preparation for the data collection effort, team members will be provided with written descriptions of past project activities and strategy documents before traveling; time should be built into their schedule for this preparatory work. Within any given country, the team may have to travel to sites outside the capital city, as necessary. A proposed travel schedule and work plan will be submitted to the USAID CTO and approval obtained before any foreign travel. Short trip reports will be submitted following foreign travel. A draft report will be written and submitted to the CTO prior to a meeting with USAID/W to present findings and conclusions. A second draft report will be prepared following comments received at the presentation meeting. The final report will contain be submitted within two weeks after the last comments are received from USAID/W and will be NTE 40 pages in length, with a two-page Executive Summary and appendices when necessary. Summary of Key Elements of the Modified SOW The assessment made the following modifications, some of which are described above, to the SOW, in respect to the limited time and resources available, and in consideration of the size of the participant population and distribution over 45 countries: • Modified the purpose of the assessment to remove bias • Declined to compare the original project goals with the outcomes • Modified the methodology so that it was more data-driven than the SOW required to rely as much as possible on quantitative evidence of impact. • Agreed that the assessment would not appraise the management of the program the performance of the contractor. • Modified the Kirkpatrick framework to fit the Africa-oriented, long-term training aspect • Agreed to try to seek out exceptional or noteworthy impact • Agreed to visit institutions in the country visits • Modified the number of site visits conducted from six to seven • Added a special team meeting in Africa, held immediately after completion of the first two site visits (Mozambique and Namibia), to review those experiences in order to make mid-course corrections in the methodology. • Agreed to focus the impact assessment at higher levels than the individual (institutional, sectoral, etc.) • Agreed not to compare the ATLAS and AFGRAD programs • Declined to conduct the visits suggested in the SOW to four U.S. universities that participated in the ATLAS/AFGRAD program due to resource limitations (as described n the report) • Modified the methodology to include a new survey tool for the Internet to find participant impact • Agreed that the assessment, despite its insistence on evidence-driven findings, cannot present scientific proof of impact. Generations of Quiet Progress The Development Impact of U.S. Long-term University Training on Africa from 1963 to 2003 An evidence-based impact assessment of the value obtained from major investments in graduate education for 3,219 African professionals by USAID and its partners in the ATLAS and AFGRAD programs Volume III Participant Text Answers to Open-Ended Questions Prepared for: U.S. Agency for International Development Bureau for Economic Growth and Trade Education Office Washington, D.C. Prepared by: Andrew Gilboy, Team Leader Harry Carr Thierno Kane Robert Torene Consultants to Aguirre International Guided and Assisted by: Cristin Springet, USAID/EGAT/ED Prepared for the U.S. Agency for International Development by Aguirre International under the Global Evaluation and Monitoring IQC, Contract FAO-I-00-99-00010-00, Task Order 13 SEPTEMBER 2004 Volume III Participant Text Answers to Open-Ended Questions A. Two Most Important Examples of Acquired Knowledge, Skills and Attitudes B. Two Most Important Examples of Applied Knowledge, Skills and Attitudes C. Examples of Changes in Institutional Output D. Descriptions of Levels at which Changes Occurred ANNEX A MOST IMPORTANT EXAMPLES OF ACQUIRED KNOWLEDGE, SKILLS AND ATTITUDES ______________________________________________________________________________ Question 37: Did you acquire any specific knowledge, skills, or new attitudes from your academic program in the United States?] If “Yes,” please indicate examples of the THREE most important skills, attitudes, or specific knowledge that you acquired from your U.S. academic program. [Most participants responded with one or two skills.] Q37a – examples Q37b – examples designing research projects supervising thesis projects at UEM conflict resolution-human relations public funds administration attitude towards work leadership skills leadership without authority knowledge about policy analysis computer skills management control system bargaining power analytical ability leadership management strategic thinking decision taking macroeconomic analysis econometrics research team work more responsibilities to my tasks broaden view of my field work learning skills research skills skills in project management skills and knowledge in information technology advanced business skills and knowledge public health knowledge professionalism self reliance assisted reproductive technologies laboratory technologies do everything on time use the best knowledge project design and implementation research skills language time management and project management skills capacity to concentrate more on reading And research acquired better understanding of general economic from electives. finance information technology diagnostics time management & analytical skills taking decision designing and using instructional materials language ability computer skills international perspective of life English self confidence laboratory and research management critical thinking information technology literacy ethics disease surveillance research conduct communication in public skills teaching education planning team work leadership skills English management and marketing skills use of info in architecture use of GIS in solving urban problem leadership initiative in doing things competence in depth analysis master accounting management management skills computer skills the importance to have vision self confidence cities management project development computer literacy counseling skill more business-like in my approach to work. self-confidence working independently taking risks/showing initiative sharpened skills of road design developed an attitude of pre-planning before events enhanced knowledge in agricultural science project management skills conduction of scientific research preparation and delivery of lectures acquired more knowledge acquired counseling skills higher technical professional skills in seed technology knowledge of the application of statistics to research work enhanced knowledge especially international perspective exposure to state-of-art facilities and leading academics computer literacy counseling skill a holistic approach to solving problems in education in most appropriate ability to teach in the university; trained more pathologists to a meticulous approach to research knowledge in quality control and its application leadership enhanced technical ability in my chosen specialty lobby and advocacy skills importance of agriculture in the developing country agricultural credit as a tool to develop agriculture plant pathologist as profession a meticulous approach to research and work specific knowledge in biochemistry scientific knowledge in general soil survey practicals aerial photo interpretation & analysis academic knowledge teaching skills confidence in reading challenges in any situation knowledge of typing and intro to computers analytical skill communication more knowledge in subject area mastery in the use of a vast scientific equipment research skills public speaking skills prestressed concrete design work ethics the level of my knowledge in my field of study has changed. I got a better understanding of western culture (respect of laws and rules). objective oriented attitudes analytical skills self-reliance entrepreneurship positive thinking adaptive management ability to carry out system analysis ability to be critical competitiveness hardworking managerial capacity writing proposal project skills in implementing multiple evaluation research studies better understanding of large international companies' activities English policy analysis manual performance chicken production economics scientific writing computer use proposal writing technical skills managerial skills scientific knowledge communication knowledge in gender and development research methods technical skills managerial skills problem analysis stating objectives English proficiency quantitative method analysis self discipline to go an extra mile to accomplish a goal knowledge of English upgrading my skills environmental manager negoiateur avec plain pouvoir leadership small business financing writing research proposals conducting a team organization improved and articulated communication skills teaching experience work ethics research skills lifelong learner regardless of my doctorate specific knowledge in virology and other health problems learned definite fundraising strategies and techniques independent and creative thinking focus university research conduct and management assertiveness use of computers information systems quantitative skills proven useful in research getting on with people entrepreneurship existence of cultural differences but also similarities being innovative to accomplish a task working hard and taking initiative research skills knowledge in HIV/AIDS research and control increased attitude to be self employed economic and finance skills can do attitude project management approach to construction research methods increase attitude to be self employed working independently by conducting laboratory tests myself attitude to distinguish between different levels of material for teaching/interaction ANNEX B MOST IMPORTANT EXAMPLES OF APPLIED KNOWLEDGE, SKILLS AND ATTITUDES ______________________________________________________________________________ Question 39: [After responding to the question of how much of your knowledge, skills or new attitudes you have applied in your work.] Please indicate which ones you selected as “a lot" or "a great deal." research skills, knowledge in public administration and professional orientation. technical knowledge increase knowledge in structural engineering, improved technology and goal oriented attitude to work. teaching under-graduates only, I do my best to pass on my knowledge to show them how to be hardworking and efficient. result oriented mind banking and finances management; project management I tried to apply a combination of all skills acquired. business oriented and simplicity started New Big Project Within The Ministry That Covers The Whole Government all three but have Ph D not curriculum development (national) and assessment in sciences leader and research skills staff-development business skills as research analyst and as accounting manager public health skills research skills enabled me to do research in Kawada planning skills changed the immunization program country-wide from outreach to station. all skills and attitudes listed have been applied by me. conducting research and supervision of postgraduate students. creativity and facilitation public speaking and research skills problem solving/analytical skills and presentation/teaching skills English unselected group follows transparency in project management macroeconomic management I can apply the theory I learnt to practical issues that I see in my daily work management research skills and language I used my computer skills to write my papers with my education skills initially things were done normally but now the use of computers are involved the way things are done in the firm has improved and my teaching techniques have increased disease surveillance education planning I introduce the teaching of new courses at the natal university of Benin In my work I have to work with groups, listen to others and enhance effective communication better knowledge of economics and more professional attitude towards work. road design skills immediately on return; effective planning and public speaking later. technical/professional skills, project management & international skills new academic perspectives and advocacy skills. knowledge in quality control helped in the setting up of the laboratory and its operation I have been teaching plant pathology in the four university & supervised over 20 MA and PhD students in plant pathology soil survey and GIS time management in organization of my program and ability to conduct research on any topics management and communication skills I teach mathematics at the university level self-reliant and computer skills adaptive management networking managerial skills supervising many chicken farm around Bamako economics and research communication & consulting besides job I am part-time teacher linking theory to practice was not easy new skills help me to be very competent in my field background in agricultural economics make me understand things applying all the above new attitudes to work hard, smart and pressure using internet and external communications to market education of stake holders. fundraising strategies and techniques, managerial and administrative skills goal oriented & analytical strategic-thinking; application of understanding of the business; assertiveness greater sense for individual difference resource mobilization & computer literacy and implementation of national student health survey current job require all the skills, knowledge & attitudes required from studies. can do attitude been involved in research; had training in short courses and plan to go for PhD has assisted me in mobilizing and counseling patients with HIV/AIDS and or Hepatitis B virus (HBV) infections. computer skills, designing and using instructional materials and research skills. grant writing skills, quantitative skills and computer skills. project management skills, entrepreneurship innovation, legal advancement and arguing for reform. Question 46: [Have you been able to apply the knowledge, skills or new attitudes acquired in your U.S. experience to other areas of your life, such as within your family, in other professional relationships, or in the community?] If “yes,” please give some examples of how you have done so. as a free law consultant (independent ) influencing local government for reasonable decision making process private consultant and teaching at the university helping others to set up their private business. bringing experience gained during my training more open to accept the difference of opinions; I make my judgment based on facts and accept freedom of expression in professional relationship; how to organize a research, how to discuss funding and organize an assessment center training became active member of the church and joined association related to with mental problems I have worked in multi-disciplinary teams to help improve neighborhood life in Nampula. consultancies I created a private consulting company and I trained various people including relatives in projects design and evaluation I became involved in working with civil society groups in research conference modeling the way & influence people on the value of self reliance and professionalism participating in consultancies and in the field work with charity org. after my return I feel that we take our life more organized, every thing is time based and also based in the advantage we will get from each step we take relation to the problems of the less fortunate in the society and the ways to improve their condition I have more interest in reading ,relating to other people's lives and I feel better organized and more motivated I'm helping my church in managing the construction of new parish in Maputa. I suggested and my family approved to save some resources to invest in the market having in mind that I acquired some marketing diagnostic skills. In my family I tried to teach the respect of time, appointment and the courage to be yourself. I created a small consulting organization named T.C.A Consulting- specialized in taxation. Continue speaking English and using computer skills and avoiding corruption in my part time teaching work & consulting work written research protocols and have a private project org. My stay in the US exposed me to another culture and my study stuff increased. Pragmatic approach of any problem I share with my children the differences between our education systems and the US. Conception of syllabus before what I have to do and time management and I do my own things well I am the president of Benin AFGRAD/ATLAS Alumni Association. learning by doing practicing what you preach and open mindedness I am very proud of myself as I share with my friends and family how life in the US is like. helped my wife to run a small business and a member of political party incline to participatory approach to community problems now in the gospel ministry and counseling people every day My work is based on changing family and community attitudes to gender relations. I live it and share it. application of planning and public speaking in my church and professional institution. have traveled with my family to academic institutions all over the world and served as external examiner and assessor for many institutions participated in workshops and conferences and used counseling skills in various situations. able to fit into any new grouping that I find myself in. served on community boards in the health and human rights sector where this new knowledge is infused. organize training programs for the youth in my church & family counselor helped me serving on various nanal committees in divers areas, different from my original line of studies I do a lot of basic household chores and repairs myself and insist on punctuality to all social and business Events. organization of my domestic affairs profession has enabled me to educate my children some now in the university abroad consultancy in soils and land use for national and international organization, extension of pedagogical data to improve individual farmers who require such services improved upon by singing as a soloist- in the US sung with the Mendelssohn club of Philadelphia The broad outlook, US program has enabled me to branch out into other areas of specialization of consulting, administration, and authorship. helped set up management structure of Ghana institute of management and other professional bodies infuse work ethics and good planning for the executive of research in the laboratory particularly with my graduate students My public speaking skills have been of use in my church and other non-academic areas. I also do mentoring of young people which is something I experienced my self when in was in the US showing how to be more result based in life in general discuss with friends and relatives; teaching people using project management in establishing wedding plan I carry out several community activities outside my job. I lead a local NGO who specialize in rural development. I provide voluntary consultancy for NGOs and institutions to help them in building institutional capacity. conference speaker (findings on my own research); creating an association gathering people sharing the same philosophy. more practical now and manage family with wife help graduate to establish themselves as entrepreneurs My children know how to use computer and the web. biostatistical consulting for professional from different sectors American University Alumni Association formed in Mali to engage in activities like social, economic & cultural improving the people share family duties with my wife trainer of NGO members, reading skills to my children and community association to consider what is good for the family, community, and the country above our own interest pre-judgment of people is not good besides my work I delivered course in development economics advice better management in association that I belong to punctuality and time management sometimes interpret simultaneously for international organizations during local and international seminars NGO consultancy and political activity member of AWLAE thru which I worked with rural community ready to share information with colleagues Things that people learn in their academic life programs will affect the way they live and perceive everything else not only at work but within their families, friends. in my own professional architectural consultancy practice; in my commercial flower farming. With my master’s degree knowledge, I have been able to revamp my personal private school called Fazo Hill College School. I have participated in the teaching about the dangers of HIV/AIDS and HBV to various communities in Uganda; have mobilized and counseled the HIV/AIDS and /or HBV patients including member of my family; have also established an NGO (Adult Literacy and Initiative for Development-ALAID) with other people in Uganda. I teach independent and creative thinking to my children. I also teach this way of living in my church. successful conception and realization of family projects; harmony in the family. I have external examined in other universities, served on national and community committees. in multi-national collaborative research projects; in national and international research or professional meetings. ANNEX C EXAMPLES OF CHANGES IN INSTITUTIONAL OUTPUT ______________________________________________________________________________ Question 48: [If you were able to apply your knowledge, skills or new attitudes at work has there been any difference in your institution's output (in terms of quality, quantity, etc.), performance, productivity or impact? In other words, did anything change?] If “yes,” please give one or two concrete examples below. more students get involved in research, a capacity building project for staff was sustained the aggressiveness of the activity implementation was higher The NGO is undergoing an institutional restructuring program in order to become more professional, the reporting practices have improved. able to contribute to the success of my org in helping to create a new business publish articles on agricultural productivity in Mozambique; on cotton and food Crops My colleagues are now more comfortable to argue with me over a specific issue. I trained my new colleagues and introduced them to the new SNA . Our work in Nampula has brought changes on people understanding of the importance of policy analysis. increased government budget pilot stage for organizational development in other government units I have started and set up a bilateral trade assistance strategy at the Mozambique High Commission in Kenya where I'm currently posted. As I was training faculty in teaching methods we saw an increased quality of learning process in target departments. quality of presented paper & applied research project planning The department of economics at ISRI started working as team having regular planning meetings. I participate in a quality committee meeting at my institution where I have I vital role . all computers are connected full time at my institution. Besides creating new jobs, we introduced periodical training programs for key personal and we were able to improve their performance and the quality of service the way of training teachers With my English skills I can easily interact with my partners in English speaking countries. Sales results and marketing plans, development and implementation. Improvement in quality work, the service is chosen between all labs to have an accreditation a balance sheet I took 2 weeks to finish now I take one day to complete it, improvement in relationship management The government has promised to provide refrigerator to some health centers and a new case of Acute Flaccid paralysis was discovered. a better result in my sector still I won't be able to give example the income in terms of sales increased info technologies, especially computer based processing makes it easier to work and archive oil production, the Senie oil field as a free lance service provider, the output and performance are judged by my clients Things change in my life and although people at my job are very jealous of my diploma, they respect me a lot and my office and salary have increased. efficiency due to computer based management of finance I designed a proposal for funding and was accepted. I succeeded in getting my colleagues buy my ideas and work as team. It was hard at the beginning but ok now. elevated to the pinnacle of my profession been able to set a factory employing about 500 people, contribution to reforms in mining sector resulting in direct private investments in the sector which in turn has led to employment of over 1500 people staff is hardworking in the organization and take initiative which has helped catapult it into a credible, viable organization that works in human rights protection for women and children application of maintenance management system in the Ghana highway authority and actively applied strategic planning in highway authority and ministry of roads and transport. changed and introduce new academic programs in my department development of guidance and counseling programs at UCC and supervised lot of higher degree students The establishment of a seed grower program rapidly increased the national seed program and the conversion to a corporate entity and paved the way to privatization. Training programs are unique in quality, content, design and presentation, the target group always mentions this in their evaluation Attitudes and work ethics did change for better and new assessment skills were developed. As a telejournalist and program presenter in Benin city, Edo state- Nigeria, the program I co-produced initially for three years. As pioneer in the customs laboratory I have contributed towards its growth from a novelty to the present status as the recognized scientific arm of the main revenue collecting agency of the government. expansion in the rural banking system have taught plant pathology to many undergraduates, some now plant pathologists Students are better trained because of my teaching and research in the department of biochemistry. coordinated multidisciplinary environmental impact on the building of the Volta dam in Ghana My institution was able to provide sound and good advice to government agriculture sector to increase crop yields and food security. At both undergraduate and graduate levels students keep on enrolling for courses in policy studies in education and skill of writing. 300% increase in profit over first four years of company (Jecty & Company LTD) Students are graduating with better knowledge in molecular biology from my department which is a great asset. My department output is more efficient because I have been able to computerize some of the work, e.g., students grades, introduced a new course in speaking skills at graduate level in my department which has improved the seminar presentation skills of students. Productivity is very high; there is very little listening at the workshop. At the private university where I used to work, students have less difficulty to get hired; because of the procedures for conferences, the company I work with gets more business from U.S buyers. respect deadline; quality of work Students know how to use the internet to look for information to write papers. increased income; more savings I could say I limit the worst effects. I was key to the elaboration of new program-outside my job- my activities have influenced several sectors. The "result oriented" state of mind is now being accepted by my subordinates, very untypical to a government organization. concern given to reports-inference on decision making; importance of the spirit of a team building. contributed to nationalization of teaching staff My NGO gets financial aid and has members trained in gender analysis. institution not only financial but provider of good ideas Research activities supported by USAID in Mali were efficient. My expertise in African marketing was very useful in price and trade analysis. better organization, i.e., improve organization quality to enhance team work I supervised thousands of students writing their thesis and I also supervised their student teaching. My English skills are very useful in my institution and new perception in environmental issues. Actually I am a consultant in 4 E conform (council formation) in business, environment study and evaluation. Research quality in our institution has improved and produces better vaccines. mainly team work but my personal contribution in planning and training skills has improved a great deal Black intellectuals began to develop as individuals and graduated to contribute to other sectors. involvement in regional workshop on education 15 classrooms were from solicitations for funding to the Namibian government foreign embassies, parent contributions to school development fund over period 2001—03 Activities/projects executed in a more focused manner. improvement in the benefits of university employees; most policies in place more elaborate business analysis; more utilization of it compilation of quality Human Capital Management programs Work efficiency in my program changed from national to district level. Sound policies guidelines and other resource materials in place. Project work is entirely outcome-based and the deliverable were all related to my field of study. SME finance and developing capacity of small and medium enterprises to become more profit and productive upon return establish ICT committee at the college and helped in the training of other students not directly but my contributions in meetings could have changed one or another view thinking. database for town planning; production of informational brochure awareness of what could be completed if you apply a structure approach to problem resolution There is greater professionalism, positive attitude, better understanding of different issues, people and cultures, better team work and broad knowledge about world affairs. All these enhance my performance and productivity. I wrote the course curriculum/program and regulations for the new course of architecture faculty of technology Makerere University; was the first head of the department even though I left before actual admission of students (my leaving shocked the university administration into seriousness and providing funds for equipment/materials to start the course). I have supervised many students who have graduated in adult education; have helped to improve performance in organizations and government through training consultancy work. Have acquired new techniques for the detection of HIV/AIDS and HBV which have assisted me in implementing some of my research proposals. Also, in rural areas where I have mobilized and taught parents about HBV infection, many women have started accepting vaccination of their children with HB vaccine. Initially there was fear that the vaccines were contaminated with HIV. Almost all public enterprises my unit has supervised have had increased revenues and profitability. I also resuscitated a development bank that had been condemned to liquidation; our organization is now more output-oriented than activity oriented. Strategic plans and vision 2025 were developed and implemented; faculty has realized "substantial" grants for research and PhD studies; establish new programs. We have trained more Ugandans in the field of instructional technology at both undergraduate and graduate levels. My institution became the most vibrant in research activities and research outputs during my tenure as director. It has one of the largest independently sourced budgets. Government funding for education inadequate. Developed strategies to mobilize resources to rehabilitate school of administration and expand it. Because of exposure and self-confidence in studio practice, young graduates become self employed. Through my training, I represented my institution as an instrumentation specialist to the national agricultural research project (NARP)- a World Bank/Ghana government project. The university's programs changed to modular course-credit form. National agricultural policy and development programs. I develop communication aspect within the institution; I share what I have learned o motivate colleagues; I also stimulate talented people to contribute in our activities; I multiply contacts to enhance our network and get resources to realize our objectives. I work within International Institutions in Mozambique and extend my specialized skills in international institutions. People at the community level started to be aware of dangerous signals of pregnancy and take decision to go to hospital. At the environment Ministry we were able to approve major laws that will enable the country to applying responsible policies of development. My institution is much more involved in international projects. quality of curricula at faculty I work and sense of community enhanced team management and better working relationship helped set up matrix of policy analysis and trained workers to apply it better expressing in team work, organization in work and capacity to express ideas laboratory management and work As director I was able to introduce the concept of management of renewable natural resource. My position at the president's office may have impact on major development policies. The NGO I am working for is taking care of the needy all over the world and my country is one of the poorest in the world and there is a lot to do. Supplying the organization with human resources and the logistics it needs to achieve its goals( school feeding humanitarian assistance, women microfinance, agriculture, etc. ). The ministry of finance is the owner of IFMS but the system is to be distributed to all government offices. Because of participation in HIV/AIDS conferences on curricula of tertiary education, a guide line content for incorporation of them are presented to the university. On personal level at our faculty(colleagues have proposed me to be the next dean) on regional level. Director in law firm and great deal of managerial and office admin duties are my responsibilities. work and develop teaching materials and serve the country in curriculum panel Climate in school change but I would like to be involved on a regional and national level. Change was mostly with my subordinates where I insisted on quality performance and accountability where before there had been little or none. The National University of Rwanda signed cooperation agreeable with Johns Hopkins University and grant from NIH under my leadership, the community leaving within the catchment area, received better care delivery. New sugarcane varieties bred by the east African community countries (Kenya, Uganda, Tanzania) had to be screened for disease resistance at the disease testing unit in Uganda. when I demonstrated the effectiveness of proper planning and community involvement the national program took that strategy At the level of project implementation-clientele were better advised on how to implement projects; nationally—the Uganda development bank acquired good status by having its own headquarters—thanks to my construction management classes . numerous institutional changes like curriculum diversity, individual output assessment and research culture; community - never to demean work, community contribution and persistent and consistency at whatever is done I have worked at the community, regional and institutional levels. Consequently, my work has impacted on all the levels. e.g. as executive director of ASESP, in 1993 I succeeded to host the international conference for social studies in Nairobi, Kenya. This brought close to 500 educators to Africa. as head of department within faculty; as teacher of students who took up responsibilities at university levels; as a member of government policies formulation My staff were highly motivated as I experienced no crisis during my tenure, cooperative societies that I worked with have survived international competition; I made contributions to regional and international for a to which I was always invited to make professional contributions. Change and outputs at the department/institution level, resulted in changes at national, regional and international level; national changes within the health sector affected the community. Result oriented management has been introduced at all levels of service delivery through output based planning, monitoring and service delivery. controlling successively cassava mealybug (1989-1995) ANNEX D DESCRIPTIONS OF LEVELS AT WHICH CHANGES OCCURRED ______________________________________________________________________________ Question 50: Please explain your answer [to the previous question about your level of change] or add any relevant information to help us understand at what level you were able to bring about change. Government funding for education inadequate, developed strategies to mobilize resources to rehabilitate school of administration and expand it. Because of exposure and self-confidence in studio practice, young graduates become self employed. Through my training, I represented my institution as an instrumentation specialist to the national agricultural research project (NARP)—a World Bank/Ghana government project. The university's programs changed to modular course-credit form. national agricultural policy and development programs I developed communication aspect within the institution; I share what I have learned o motivate colleagues; I also stimulate talented people to contribute in our activities; I multiply contacts to enhance our network and get resources to realize our objectives. I work within international institutions in Mozambique and extend my specialized skills in an international institution. people at the community level started to be aware of dangerous signals of pregnancy and take decision to go to hospital. At the environment ministry we were able to approve major laws that will enable the country to apply responsible policies of development. My institution is much more involved in international projects. The quality of curricula at faculty I work in and sense of community enhanced. Team management and better working relationship Helped set up matrix of policy analysis and trained workers to apply it Better expressing in team work, organization in work and capacity to express your ideas As director I was able to introduce the concept of management of renewable natural resource. My position at the president's office may have impact on major development policies. The NGO I am working for is taking care of the needy all over the world and my country is one of the poorest in the world and there is a lot to do. supplying the organization with human resources and the logistics it needs to achieve its goals( school feeding humanitarian assistance, women micro finance, agriculture, etc.). The ministry of finance is the owner of IFMS but the system is to be distributed to all government offices. Because of participation in HIV/AIDS conferences on curricula of tertiary education a guide line content for incorporation of them are presented to the university. On personal level at our faculty (colleagues have proposed me to be the next dean)on regional level. Director in law firm and great deal of managerial and office admin duties are my responsibilities. Work and develop teaching materials and serve the country in curriculum panel Climate in school change but I would like to be involved on a regional and national level Change was mostly with my subordinates where I insisted on quality performance and accountability where before there had been little or none. The National University of Rwanda signed cooperation agreeable with Johns Hopkins University and grant from NIH under my leadership, the community leaving within the catchment area, received better care delivery. New sugarcane varieties bred by the east African community countries (Kenya, Uganda, Tanzania) had to be screened for disease resistance at the disease testing unit in Uganda. When I demonstrated the effectiveness of proper planning and community involvement the national program took that strategy. At the level of project implementation—clientele were better advised on how to implement projects; nationally—the Uganda development bank acquired good status by having its own headquarters￾thanks to my construction management classes Numerous institutional changes like curriculum diversity, individual output assessment and research culture; community—never to demean work, community contribution and persistent and consistency at whatever is done. I have worked at the community, regional and institutional levels. Consequently, my work has impacted on all the levels, e.g., as executive director of ASESP, in 1993 I succeeded to host the international conference for social studies in Nairobi, Kenya. This brought close to 500 educators to Africa. As head of department within faculty; as teacher of students who took up responsibilities at university levels; as a member of government policies formulation. My staff were highly motivated as I experienced no crisis during my tenure, cooperative societies that I worked with have survived international competition; I made contributions to regional and international for a to which I was always invited to make professional contributions. Change and outputs at the department/institution level, resulted in changes at national, regional and international level National changes within the health sector affected the community. Result oriented management has been introduced at all levels of service delivery through output based planning, monitoring and service delivery. Controlling successively cassava mealybug (1989-1995) Q50b. level you were able to bring about change responses New lab and equipment for research were procured For example in policy matters related to microfinance regulation in Mozambique I presented to the NGO members, board and management assessments of the organization’s performance and proposals for new approval. Creating new entrepreneurs The studies and in fact evaluation improved a productivity agricultural area of Mozambique. The team work that led to the establishment of school of communication and arts has great impact within the university. Change in attitude towards concepts or environmental health and safety at working place New colleagues with non experience in job were able to understand their job after the specific training. More people especially In Nampula province are taking advantage Of awareness of market opportunities of agricultural communities. at the level provincial directorates of agriculture Skills I applied within the institution I worked, improved the quality of services rendered to the business community involved in bilateral trade between Mozambique and Kenya. Prepare and administer training courses in the field and other research work undertaken under my control Working for World Bank finance-project and one of the major problems was having things done on time At the department level the changes particularly evaluation of activities were resisted beyond the level of department. Innovation Due to lack of resource it is still not a relevant change Any relevant information at the institution can be sent using Internet my computer is connected to Internet. Within the institution (see comments on Q48) and in the community I have created new jobs improving the lives of people. I am working with the pilot of training of MEPS. He is the final person that can take decision about trainings ( network ). Marketing and financial department Member of south regional expert group for the rewriting of manual about 'harmonized methods of food products and analysis With my firm customers understanding has improve in accounting and tax system. The strain of polio discovered on Chad came from North Nigeria. A big immunization is being conducted in Chad, Niger and Benin now. I think that by applying the knowledge and the skills The number of customers have been increased and sales also changed. I am involved in oil exploration and exploitation works and promotion of coastal sedimentary basin of Benin. Client institution and collaboration Regular control over local government helped my department find out administrative and finance problems that needed to be addressed at local level. My contribution brought change to the mining industry in Ghana. I became an authority in road management and planning in Ghana with several papers presented at international conferences. Biochemist program at the faculty of science and medical school and faculty of agric at the university were all enriched. My publications and papers have helped to increase knowledge and competency within the profession. I was requested by government to work towards converting the purely civil service entity into a company. The changes were part of the process, which were accelerated when I became CEO four yrs after my return. I work within the community where the target group is mainly rural chiefs and others now accept the message of gender equality. Within the medical school and health sector In both media and educational institutions, the minds of people are affected by information. If the information is useful positioned and constructive, attitudes change not just within the individual but in the larger community. Contributed towards establishing a new unit in my inst which enhanced its activities that impacted positively on government revenue Head of rural banking department The institution is responsible for advising government as agricultural matters and food production. Teaching at undergraduate/ graduate levels and participation in national and international levels and authorship of a book for international use Within the institution students have been trained and these are employed in various institutions and equipment in Ghana and elsewhere. My organization recognizes the importance of allowing students to use the Internet to look for information in research. My work focuses on policy analyses and program for environment and rural department- I carry out several community activities. Every information (reports) is important and could be the major ingredient to strategy control (this was the message I want to share to the staff). Production of vaccine which are sold All my presentations of programs at annual meetings are presented in power point Women I trained are in associations Targeted institution to support was more renown in presenting activities to be supported by USAID My foremost contribution was in this area of tools and techniques for price and trade policy. Many former students are teaching English in Malian high schools or working in NGOS. New perception of my consultancy on environment and help NGO and negotiation during international meetings As a trainers of trainees I am responsible for training of teachers increasing rate of literacy in the country. MP'S are now skill thru the institution and applying themselves at the constituencies’ level. Planning regional workshop on education (southern Africa) and special education on radio talks Following school became keener to self-improve infrastructure; level of commitment increased and loyalty of staff increase; community contributed to build classroom something difficult to achieve Improvement in employees means we are able to compete and retain not national but international. Working in the department for a year due to move to another department soon. On corporate level Persuade supervisor and donor agencies that a vital program is undertaken like any other donor funded program, that there is a need to set up a national surveillance system for monitoring youth issues for better. Being the chairperson of the staff development committee, it is very easy for me to implement some new ideas. I brought change—mostly in my interaction with my students. The project I worked in left a remarkable change in the education technological field on community level and nationally in the education sector. At institutional level—development and institutionalizing a new organization at sector level the number of small and medium enterprises supported has impact in the sector—bringing a low growth. The college opened its doors to the public. I ended serving in the national panels, ICTs, curriculum, etc. Work in the legal practice depend very much on what instructions the clients bring to the firm. Having centralized database of keeping records for subdivided, consolidated, incorporated info brochures accessible to the public Created institution that could serve as a platform to unify stakeholders in transport sector to cooperate to the benefit of SADC region. At the moment, my input has been bringing a better understanding about African problems, challenges, economic and political issues. I have taught university students who graduate and serve the country in the field of adult education; carried consultancy at district and organizational level. Collective and constant teachings about the consequences of unprotected sex have brought about behavioral change, particularly among the youth, which has led to a significant decline in the HIV/AIDS prevalence—Uganda. International, the World Bank has advocated the Ugandan model of parastatal monitoring to Malawi and Rwanda Our unit has also given presentations to three other African countries. New programs and funding at the institution, nationally through consultancy/contract employment, internationally through research collaboration with other universities I started a masters' degree in instructional technology in addition to teaching at the undergraduate level. My institution restructured to a larger autonomous unit within the university. Through national and international linkages it received acclamation for its work in training and research.