AVTPAWD Mid-Tcnn E\aluation Rcpon P()-R bj -5 Mid-Term Evaluation Report World Concern Development Organization (WCDO) Adaptive Vocational Training Program For Adolescents with Disabilities Vietnam June 2003 AVTP:\WD \lid-Tsnn Evaluation Repim Table of Contents INTRODUCTION ......................................................................................................... 4 PROJECT BACKGROUND ............................................................................ 5 Project Summary ................................................................................................ 5 Project Goals and Outputs .................................................................................. 5 EVALUATION METHODOLOGY ............................................................... 7 Rationale ............................................................................................................. 7 Constraints .......................................................................................................... 7 Sources of Information ....................................................................................... 7 Sampling and Selection ....................................................................................... 9 Total Number~Type of Interviews Conducted .................................................. 10 Budget and Asset List ....................................................................................... 11 COOPERATIVE RELATIONSHIPS ........................................................... 11 National Events ................................................................................................ 12 Disability Forum ............................................................................................... 12 Referrals to Cooperating Agencies ................................................................... 12 PROJECT MANAGEMENT ......................................................................... 13 Operational Environment .................................................................................. 13 Implementation Schedule .................................................................................. 14 Project Design ................................................................................................... I5 Project Management Structure .......................................................................... 15 Project Beneficiary Selection ............................................................................ 19 Baseline Situational Analysis (BSA) ................................................................ 21 Training Needs Assessment (TNA) .................................................................. 21 . . Participation and Monztonng ............................................................................ 21 Advocacy .......................................................................................................... 22 PROJECT OUTCOMES ............................................................................... 23 . . * Awareness Raising ............................................................................................ 23 . . Adaptive Vocational Tralmng ....................................................................... 26 Employment and Income Generation ............................................................... 29 Supportive Networks ........................................................................................ 31 Formal and Informal Training Approaches ..................................................... 33 SUSTAINABILITY ........................................................................................ 36 REPLICABILITY ...................................................................................... 37 RECOMMENDATIONS ...................................................................................... 37 * ANNEXES ..................................................................................................................... 43 June 2003 Pagc 2 :\\l'P\\VD E\tiuxinr! ,\VTPAWD Mid-Tcmi E~~luarion Rcpon List of Acronyms/Abbreviations ABE AoG AR AVTPAWD AWD B A BSA CBR CBSW CEDC CPCC CR CSW CWD DCOF DELISA DET DOH DSC EMW GSRV Hvo JSC KAP MOLISA MoU NWMT P.C. PACCOM PC PMT PRA PVO PWD RC RCV SWTG TNA TOR TUC US AID VNAH VM> WAF WCDO YDA Alternative Basic Education Assemblies of God World Relief Awareness Raising Adaptive Vocational Training Program for Adolescents with Disabilities Adolescents with Disabilities Blind Association Baseline Survey Assessment Community-based Rehabilitation Community-based Social Worker Children in Extremely Difficult Circumstances Committee for the Protection and Care of Children Country Representative (for WCDO) Community Social Worker Children with Disabilities Displaced Children's and Orphan's Fund Department of Labor, Invalids and Social Affairs Department of Education and Training Department of Health District Steering Committees East Meets West Government of the Socialist Republic of Vietnam Health Volunteers Overseas Job Service Centre Knowledge, Attitude and Practice Test Ministry of Labor, Invalids, and Social Affairs Memorandum of Understanding Northwest Medical Teams People's Committee (GSRV) People's Aid Coordinating Committee Project Coordinator (WCDO position) Project Management Team Participatory Rural AssessmentIAppraisal Private Voluntary Organization People with Disabilities Red Cross Red Cross Volunteerls Social Work Training Group Training Needs Assessment Tenns of Reference Trade Union Centre United States Agency for International Development Vietnamese Association for the Handicapped Vietnamese Dong (official currency: 15,000 VND = USD 51.00) Vietnam Veterans of America Foundation World Concern Development Organization Youth Disability Association INTRODUCTION The mid-term evaluation comes over half-way through the 6-year life of World Concern's Development Organization's Adaptive Vocational Training Pro,gam for Adolescent's with Disabilities (AVTF'AW'D). Program activities commenced in October 1998; WCDO's USAD ,gant is scheduled to end in December 2004. Evaluation fieldwork was undertaken over a 10day period, 22 May - 31 May 2003. According to the detailed Terms of Reference (TOR), the evaluation team was to review: project implementation, participation and monitoring, outcomes, and sustainabilityl. The evaluation process consisted of both qualitative and quantitative data collection. Questionnaire surveys were administered to the major Government partners; semi-structured interviews were conducted with World Concern staff as well as with beneficiaries in their homes and businesses; and the evaluation team reviewed project documents'. The major reference tool for organizing the evaluation was the program proposal, and specifically the statement of objectives and indicators outlined therein. Data collected during the evaluation indicated that the on-going project is being implemented in a manner that obviously benefits the AWD, their families, and community; however there are some areas for improvement that could greatly enhance potential for positive outcomes. It is anticipated that findings tiom the evaluation will contribute to adjustments of strategies to help secure employment and income for Phase Two. In general, the project is being implemented slowly, but with remarkable output considering the environmental constraints and the unique target population. The extent and level of participation varies depending upon the stakeholder: generally, the more directly and closely associated to AWD a stakeholder is the more they participate in the project and are seen to reflect Program philosophy. Despite the slow pace of the project the outcomes are essentially on target with the proposal, and were visible in ternls of income and employment, social integration, and attitudinal change. Significantly, the evaluation results sugest that the model used for raising awareness, vocational training, employment and income generation, social integration and support for AWD is valid for the semi-rural environment of Vietnam. Particularly, the use of the non￾formal sector for vocational training and employment was found to be effective and should be regarded as an innovative approach that could help the economic future of PWD in Vietnam and have a positive influence on related policies. Conclusive statements about sustainability are harder to draw at this point. It is clear that many people have been exposed to AWD and related issues through this project and self-reporting indicates related changes in behavior. Government officials, especially in the more rural areas seemed enthusiastic about what they had learned and the influence of the project on communities. However, there is yet no \isible government￾related structure or movement toward assuming responsibility for the project activities, nor any indication that activities would simply be absorbed by the government agencyiies responsible for PN'D at the cessation of the project. So in this limited sense of the concept of 'sustainability' (ie. continuation anb"or replication of activities beyond project life and external funding), the project is not sustainable. Ho\vever, if a broader notion of 'sustainable' is applied, in the sense that benefits and increased awareness will continue and will shape the future behavior of people in the target area then the project can be considered sustainable to a limited extent. ' See Annex 1 for eom~lete TOR Major project stakeholders include: USAID, WCDO, PXlT(and thmugh the PMT: DELISA), DSC (and thmugh DSC. the Red Cm). CSW AWD, AWD families, panicipating formal mining instihdufes and informal businesses, and Ihe sunounding community. June 2003 Pagc 4 .-\\'TP.-\\YO b:~iuxioa AVTP4U'D \lid-Tenn E\aluation Ruport 1.0 PROJECT BACKGROUND 1.1 Project Summary In Vietnam, adolescents with disabilities lack access to effective vocational training programs that enable them to address their own social, vocational, and economic needs. Compounding this problem is the general lack of employment opportunities, low levels of community awareness, and few opportunities for AWD to join with others in understanding and promoting their value to society. World Concern Development Organization (WCDO) is a US-based Christian international humanitarian organization that has decades of experience implementing projects in small-scale family credit and providing livelihood and vocational training opportunities for youth in difficult circumstances. Since 1998, WCDO has been implementing a USAID-funded project in Vietnam called "Adaptive Vocational Training Pro-m for Adolescents with Disabilities" or AVTPAWD. The overall aim of the project is to enable targeted disabled adolescents in Hai Duong, Danang, and Quang Nam Provinces to address their own vocational, social, and economic needs. The Program has hvo distinct phases. Initially the project was approved for a 2-year period (October 1998- December 2000) with $875,344 direct funds from a sole donor, USAID. Then USAID authorized a one-year extension (October 2000-September 2001) with $250,000 extra funds. World Concern thus regards the period of October 1998 - September 2001 as Phase One. A second proposal for a further three years of implementation was approved by USAID in October 2001, covering the period October 2001 - September 2004 which World Concern regards as Phase Two. 1.2 Project Goals and Outputs The proposal states that the project is to target adolescents (ages 13-18) with disabilities (AWD) who can potentially integrate into society, undertake vocational training, and secure employment from skills gained in training. Overall the project has 4 broad objectives: 1. Awareness Raising 2. Vocational Training 3. Employment and Income 4. Support Networking Complementary to these components are activities that include peer group development, life-skills training, family and community education, and physical supports directly or indirectly for the AWD. The four objectives form the basis of 4 principal goals of the project. These are outlined below with associated indicators of expected output. These indicators focus on quantitative outcomes such as numbers in training, numbers employed, numbers of those exposed to education, and socialiphysical adjustments to accommodate the needs of AWD'. The evaluation served to further develop these quantitative outcome indicators, as well as identify and query some additional activities and output indicators stated in the proposal4. For more detailed information on the specific mulls and outcomes tested for. sa Annexes 2 and 3 'A full Loeieal Framework Anat~is fLFA) w3s not included in the orizinal vmmsal: on the conw. Ihe vrd conlained mo~ than one liw of in 1-2 years; outcome in 3-5 y-; and impact over a 5-10 year period. Achievement of impact dew on my mvimnmenwl \Jriablts bond lhs conrml of any single development agency. Summary of the Four Major Project Goals and the Expected Outputs Goal 1: To raise awareness about disability issues involving the AWD themselves, their families. WCDO staff, government project partners, vocational trainers, ABE teachers, and the general community as a whole. Output Indicator 1.1: At least 7040% of participants in A~vareness Raising (a) training demonstrate increased awareness of disability issues. Goal 2: To undertake a vocational training project involving both formal (institutional) and informal (small businesses) participants ensuring that the AWD gets high quality training to enable them to graduate and secure employment. Output Indicator 2.1: At least 75% of Am ivho participate will s~rccess/irl!~~ complele their course/s. Goal 3: To provide access to employment and income generating opportunities for -graduated AWD, preferably within their own local community. Output Indicator 3.1: At least 70% of graduates are meeting their ~~ocational needs. Output Indicator 3.2: At least 70% of graduates are earning incomes greuter than that before the training. Goal 4: To teach valuable life-skills and help integrate the AWD beneficiaries more into society through social activities and supportive nehvorks. Output Indicator 4.1: At least 70% of graduates feel they receise greater m~rtrral supporl than before the progranl. Output Indicator 4.2: At least 70% of graduates feel that thev apply new howledge and skills. All the expected outputs of these four major goals have been met. The specific results for each of the indicators are summarized in Annex 2. Details on lower level aims and achievements are summarized in Annex 3. A\'TPA\\:D \lid-Tcnn E!.hxion Rupon 2.0 EVALUATION METHODOLOGY 2.1 Rationale The AVTPAWD is solely funded by USAID and commenced in October 1998. Initially the project was approved for two years (Oct/98 - Dec/OO). Then USAID granted a one-year extension (Oct100 - Septi02). A further three years of the project was granted in October 2001 thus rendering the period October 1998 - September 2001 as half of the total project period. Therefore this evaluation of the first three (3) years of implementation (Oct/98 - Sept/Ol) is regarded as a mid-term evaluation. The evaluation was conducted quite some time after the end of Phase One to get more of a true picture of project impact, especially in the area of training and employment sustainability. A drawback of this timing was that the project's Second Phase had already begun and the interviewed stakeholders did not always make a clear distinction behveen activities or results from Phase One vs. Phase Two. Project staff, Government staff, and beneficiaries regard the entire project as a 'seamless whole'(including World Concern's presence in Danang through the Displaced Children's and Orphan's Fund (DCOF) prior to the commencement of AVTPAWD). 2.2 Constraints During Evaluation One constraint experienced during the evaluation was that many of the WCDO staff who were involved in Phase One have now moved on. In Hanoi, only the driverltechnical assistant has been in the project since October 1998. In Danang and Quang Nam both Project Coordinators have been in the project since October 1998. Unfortunately previous project management was no longer in-country and thus not available for interviewing. The most useful information about Phase One came from the Project Coordinators. The evaluation team interviewed an exemployee of WCDO in Hanoi who was the Hai Duong Project Coordinator for the first phase, as well as the two current coordinators in the Danang Office. Other useful information came from the social workers, one in Hanoi office, one in Danang office. However both these social workers joined WCM) half-way through the fust phase of the project so their input, though important, supplied supplementary information. The second major constraint was in using project staff to conduct the interviews with project beneficiaries: CSW's were utilized as the enumerators to administer the questionnaire to AWD's and to A\VD families. WCDO attempted to ensure that no CSW interviewed beneficiaries from their own district but this did not always happen because of the low number of participating CSW's and the need to collect data xithin a limited period of time. Also WCDO staff were used as translators for the English-speaking members of the evaluation team; again, perhaps shaping respondent's answers to the questions and affecting transparency. Time was a third major constraint. Because the evaluation period was relatively short, it was impossible to provide comprehensive training for the enumerators and this obviously affected data quality. For example, there was a great deal of difficulty with obtaining good information on the ranking questions because the enumerators themselves did not understand the concept very well. A fourth constraint was incon~plete project records and especially the absence of baseline information to allow for quick con~parison of chan, ue over time attributable to the Project interventionls. 2.3 Sources of Information The evaluation utilized three main sources of information: quarterly reports to the principal donor. USAD; records kept of project components by WCDO staff and their government partners; and the inteniew with project stakeholders specifically designed for the evaluation. Id la id trl I& Lid kid d Lsl u Lpi Y Y id U Ira id .A\'TP-\N'D \lid-Teno E\~luaiion Rqwn 2.3.I Quarterly reports Twelve quarterly reports were written between October 1998 and September 2001: all were available in the WCDO off~ce, Hanoi. These reports gave updates on the principal components of the project covering: Program Management Coordination; Strengthening Cooperative Relationships: Raising Disability Awareness; Community Support Workers and Group Fonnation: Adaptive Training and Employment; Alternative Basic Education; and an Analysis of the Quarter. While the information presented gives a good overview picture of activities during the first three￾year phase the reports are, on the whole, strongly anecdotal and do not focus on the quantifiable targets set-out in the proposals. Data was not presented consistently, making it difficult to track progress against implementation plans. 2.3.2 WCDO Records On-going records, usually monthly or quarterly up-dates, have been kept at each of the WCDO project sites. These records include: national and provincial MOUs; provincial and district bank transfers and expenditure; local staff agreements and stipend records; surveys and data; AWD and family details; AWD medical and surgery records; AWD training, employruent, and income information; institute and business details; peer group records; and equipment and supply records. WCDO staff andlor the CSW's collect raw data and records are updated reglarly after site-\isits. Some sections of the evaluation interviews were designed to cross-check the validity of some of these records. The evaluation team leader checked for the existence. filing, presentation. and accuracy of records in both WCDO offices as well as at the Red Cross ofices when inteniewing DSC's and CSW's. 2.3.3 Development and Administration of Interviews and Questionnaires The Phase One proposaVs outline the expected project management system, the implementation strategy, a number of specific and measurable goals in the form of project objectives and activity outcomes, and an itemized budget. Therefore, the broad purpose of the evaluation inteniews was to gather information on these project components by interviewing the relevant stakeholders. This collected information was then used to verify data gathered and provided by staff. to confirm the implementation of stated project objectives, and to check the efficacy of implementation. The interview forms and relevant questions were constructed around an Evaluation Framework to ensure that all project objectives and activity results/outcomes stated in the proposal could be tested and verified5. The questionnaires contained major sections representing the four major components of the project: awareness raising; vocational training; employment and income; and supportive networks; as well as sections for recording demographic information and overall strengthsheaknesses. Eight different questio~aires addressing the major stakeholders were developed in English and then two were translated into Vietnamese (AWD Inteniew and AWD Family Interview) to facilitate administration of them by Vietnamese-speakers. In the case of AWD. AWD Family interviews, and Businesshstitution interviews the questionnaire was adhered to very strictly - in other cases (ie. for WCDO staff and for Governlent officials and CSW's) it was used more as a guideline. Although ori,&ally intended, the evaluation team could not conduct random inteniews of community people due to security and time constraints. In addition, there were no group inteniews as there was not enough preparation time to arrange the difficult logistics for brin-Gg the groups ' See Annex 5 for Evaluation Framework. June 2003 Paze S -\!'TPXl'?D E\~!uxim together. And due to the timing of the evaluation, it was not possible for the evaluation team to observe a regular Peer Group meeting although this had been requested. PMT, DSC, CSW's, WCDO staff, and businesses were interviewed by the hvo English-speaking evaluation team members using translators. The AWD and AWD family interviews were conducted primarily by the CSW's and WCDO staff (CBSW's participated as well). 2.4 Sampling and Selection 2.4.1 Project Management Team (PMV and District Steering Committee (DSC) It was only feasible to meet with the head of the PMT for each of the three provinces. As each head was invariably the vice-director of DELISA, the interviews were undertaken in the DELISA head￾quarters. Other DELISA staff, including PMT and DSC members, were sometimes present. Although far from ideal, this was the PMT head's choice and the interviewer was not in a position to request an alternative approach. However this was not too much of a conflict in maintaining the integrity of the interview, as the head of PMT answered most of the questions without comment or referral to anyone else present. Each interview was conversational, and conducted in English with translation to get an overview of the PMT's role, the relationship between different project stakeholders, and their perception and opinions about project development and effectiveness. The comments were recorded on an interview sheet. However, the interview sheet was not used how it was intended as most head of PMT were not interested in answering questions and preferred to give information in their own order and manner. The interview form provided only a guide to ensure that all project conlponents were covered adequately. A similar approach applied to the District Steering Committee. The heads of the District Steering Committees were invariably DELISA members; however, most DSC interviews were with the Red Cross members as this organization is much more involved in project implementation. 2.4.2 Training Centres - Formal and Informal Institutes (Formal): Formal Training Centres (government-run institutions) were responsible for training 190 AWD (36 percent of overall training) in the first phase6. The head of each training centre and when possible, a key teacher were interviewed to get an oveniew of training activities. AWD participation, and outcomes. Site inspections were conducted to gauge the level of appropriateness for training AWD, any building modifications that may have been made for better access and use for PWD, and the appropriateness of the types of training the institute offered. It should be noted that even prior to WCDO's project, the government training centers have had an on￾going mandatelquota for placing and training PWD. Businesses (Informal): Small businesses provided the bulk of the training and employment for this program: 329 AWD (64 percent of the total ) were trained in informal businesses. The evaluators interviewed some businesses that trained and employed the AWD, and inteniewed some that provided training only. After an interview with the business owner, a site inspection was undertaken to see if the building was appropriate to train the AWD, whether the type of training was appropriate, if site improvements had been made to accommodate AWD needs, and whether project equipment had been bought and maintained. Of these, 6 percent or39 had to be rewined when placed in a bushes en,. ~tmnment. .AVTP.\L\>D Mid-Tonn Eduarion Rcpm 2.4.3 Co~nmunity Social Workers (CSW's) The CSW are the principal, grass-roots, implementers for this project. Their role includes: representing the DSC; the interface behveen the beneficiaries, other stakeholders and WCDO; surveyors, data collectors, and reporters; monitoring AWD trainers and trainings; monitoring A\YD employers (businesses), employment and income levels; support for AWD family; organizers of community events for AWD and undertaking awareness raising initiatives. As their role was so diverse and critical to the success of the project the inteniew of the CSW was regarded as providing essential information about project management, implementation, monitoring. and outcomes. However, due to logistics and time constraints, as well as the CSW being involved in the AWD and family interviews, it was not possible to interview all CSW's. Thus, inteniews were conducted with one key CSW per district. Much of the information was used to verif?. information provided by upper-level management interviews, AWD and family responses. trainers and employers, as well as the collected report data in the WCDO offices. 2.4.4 Adolescents with Disabilities (Am) and A WD family Originally, WCDO aimed to randomly select and interview 20% of beneficiql AWD's and their families to provide a valid and significant sample size. About 25 percent were selected using random selection tables, to allow a 5 percent buffer for bad or missed intenfiew. The project coordinators took the list of selected AWD and revised the number downward according to availability and accessibility to the evaluation team. In the end the evaluation team inteniewed 19 percent of all families and 21 percent of all AWD. Those interviewed agreed to an inteniew either at their homes, workplace, or a designated place for convenience. This latter strategy was used mostly in Quang Nam where access to each AWD would prove logistically difficult so it was decided that they, and a key family member, would be transported to a convenient interview location (such as the Red Cross office). The family members to be interviewed were those associated with the AWD selections. It was a prerogative that only a key family member (preferably mother or father) would be inteniewed and they were done so usually straight after the AWD interview if not in conjunction with if there were enough interviewers. 2.5 Total Number and Type of Interviews Conducted %total beneficiaries. 19% 21% AVTPAWD Xlid-Tcnn Evnlun~ion Rcporr 2.6 Budget and Asset List To assess whether some of the budget areas had been addressed appropriately, a few line items were selected and checked either through the interview process or by physical observation. For instance, the evaluation team spent some time ascertaining whether or not there had been appropriate expenditure against the following detailed line items: vocational training equipment, modifications to formal and ABE mining sites. revolving equipment and supplies scheme, small-scale credit for AWD households, transport for CSW's. salary for social work interns, funds for PWD teacher training and for paying vocational mobile trainers. and income generation support to CSW's. It was not the responsibility of the evaluation team to conduct a financial audit - but spot checks of physical materials were conducted to provide some indication of whether money had been spent on what was originally proposed and to corroborate activity report's. There was nothing to indicate that money has not been spent as it was proposed (i.e. easy verification such as with checking modification to buildings and purchase of equipment). This mini-review did indicate, however, that in the project planning inadequate attention was $ven to rationalizing the budget with activities. In some cases, when asked whether or not they had conducted activities for which there were funds allocated, the Project Coordinatorls indicated that they had not been aware that they could spend money on those items or they were not aware that those acthities were ever intended to be implemented as part of the project. In other instances, the Project Management deliberately determined that a particular activity was not feasible (given the short implementation period. lack of technical expertise or experience within WCDO, or for other reasons) - for instance, the recruitment. training, monitoring, and support of mobile vocational teachers - but these decisions and rationale for them were not captured in writing and not necessarily reflected in on-going planning as more funds for some such activities were requested at the time of extension. The budget makes reference to Goods-in-Kind (three containers valued at USD $50,000 each) as WCDO's contribution to the project. Receipts and details were reported against in quarterly reports, to the satisfaction of the donor. For the purposes of the evaluation, the team was interested to know if the GiK made a positive impact on the project area or relationships, and how it was managed given the multiple lei-els of administration. The proposal did not contain information about what the contents would be used for, nor a plan for distribution. Project Coordinators and documentation did verify that hvo shipments were received during Phase One. and the contents given to PMT in all three provinces and DSC for distribution, with verbal instructions that some should be allocated for AWD. Medical equipment was donated to local health centers. However. no great effort seems to have been paid to how distribution could have had a positive impact on the pro-gam as a whole. And as no distribution record exists, it is impossible to determine the impact of the shipmentk on the project beneficiaries. Generally, officials were reported by PC's to have been appreciative although they recognized that not all the supplies were of optimal quality and that there were some issues of appropriateness both in sizing and in type of supplies. In future such concerns (which could adversely affect key relationships) could be minimized through having greater consultation with Project Coordinators about the type of supplies and resources to be provided, and a more deliberate methodology for distribution. 3.0 COOPERATIVE RELATIONSHIPS Information about cooperative relationships (defined as WCDO's interaction with national and international non-government Organizations) was collected through interviews with government partners. with project staff, and through a review of project documentation. In all cases, WCDO maintains healthy on-going relationships with some agencies although there is not necessarily regular interaction. In addition to formal organization linkages, the evaluation team assessed levels of participation by other stakeholders. Participation by DSC and community members and business owners was found to be greater in more rural and remote areas. There was not much authentic grassroots activity outside direct Project activities at any of the Project sites - what occurs is primarily a result of project influence or a directive from the central government. 3.1 National Events As reported in Quarterly Reports, WCDO staff were actively involved in organking for and participating in various national events related to the Disabled, including a Seminar on Employment of Persons Disabilities (Danang, August 2000) and the Asia Pacific Disability Forum hosted in Hanoi (December 2001). Obviously. involvement in these events was good both for WCDO's relationships with the respective Government partnerls as well as for promoting awareness of disabilities. Several DSC members and CSW's indicated that one of the variables that has made them interested in disabilities is the interest taken by a foreign agency - it is difficult to quantify how much of a factor this is in promoting interest and intewention but it cannot be overlooked as one impact of WCDO's involvement. 3.2 Disability Forum WCDO expatriate staff were instrumental in forming the Disability Forum in 1999, a cooperation and communication network that meets every-other-month and publishes a regular newsletter. WCDO staff continue to participate in the Forum and to make occasional presentations at meetings as well as contributions to the newsletter. While this group provides an excellent place to disseminate information about this project and to discuss common issues, more could be done by WCDO to capitalise on the possible benefits of such a group. For instance, deeper level of analysis by WCDO project results, and presentation in accessible form. would make an excellent contribution to national-level discussion, perhaps influence the practices of other agencies who see the effectiveness of the informal training and job placement approach, and perhaps even policy. 3.3 Referrals to Cooperating Agencies There are several Organizations with whom WCDO cooperates to discuss project implementation experiences, as well as to provide appropriate medical interventions for project AWD. The referral system is not systematic, hut largely informal as there is no project funding available to support it. In most instances, WCDO's contribution consists of logistical support for the AWD. These cooperating agencies include: ( Education. Norrh\vest Medical Teams (USA) I Medical check-up for all AWD. Pearl S. Buck I Discussion on plan to extend work for hearing Agency Assemblies of God World Relief (USA) Kommitee Twee (Holland) Type of Cooperation Funding for surgery interventions. Hearing tests for AWD @ Center for Special Save the Children UK (SCF-UK) Southeast Asia Relief June 2003 Pase 12 AVl P..\\VD E%.,)i::::~io:t impaired in Phase Two. Provided training in awareness raising for families of AWD. Allocate funds to pay for minor surgery for Vessels of Mercy Vietnam Veterans of America Foundation (USA) World Vision AWD's. Provision of hearing tests and hearing aids to AWD. Health exams and provision of prosthetic and orthopedic interventions. Sharing experiences and discussin% implementation. A\'TP!\\\:D \lid-Term E\aluation Rcpon There is some evidence that local government counterparts are learning from WCDO's interventions in this area: during the evaluation the team leader visited a home where a former project beneficiary was recovering from surgery on her club feet. The local Red Cross had become aware of her condition and used resources from the American Red Cross to pay for the surgery. Such examples could be interpreted as indicators of sustainability and increased initiative from local players. 4.0 PROJECT MANAGEMENT Project management, structure as well as implementation, is to quite an extent proscribed by the GSRV: as with all INGO work in Vietnam the management and implementing partners are government or para￾government organizations. The national government partner (MOLISA) signed a general MOU with WCDO to allow implementation of the AVTPAWD project in three provinces through their provincial counterpart DELISA. It is through DELISA that the provincial MOUs and project proposals are signed. The People's Committee of each province formed a Project Management Team (PMT), which is headed by DELISA, to oversee the implementation of the AVTPAWD through District Steering Committees (DSC). This makes for a rather cumbersome administrative structure - although in practice there is little si,gificant input from the PMT level and the DSC is more of an information link between the CSW and WCDO than actually implementing. Furthermore, location in three provinces in two major geographic areas added some interesting implementation and management challenges; although it also allowed for comparison of variables affecting how such a project can work. 4.1 Operational Environment In addition to its National Memorandum of Understanding (MoU) with the Ministry of Labor, Invalids. and Social Affairs, WCDO signs operating agreements for projects at the provincial level. For the -4VTPAWD each provincial MoU carried five signatories: WCDO, DELISA, DET, Red Cross. and the People's committee7. There was no significant deviation from the terms and conditions outlined within the h1oU except that in all cases the number of actual beneficiaries was considerably more than the fi-gre contained in the Agreement. The Red Cross members of the DSC who were inteniewed during the evaluation indicated that there was some tension between the Red Cross and DELISA at both the PMT and the DSC level - but that it pertained primarily to confusion over roles within the project rather than to supercediig or historic tensions. Traditionally, different government agencies do now work together - not least because that would require sharing limited resources. In this project, it appears that DELISA is largely a silent partner and that the Red Cross people do most of the work. The MoU does provide a clear distinction behveen the two entities as far as project responsibilities are concerned. World Concern's decision to partner with Red Cross proved to be an excellent choice. In the two provinces where this relationship is most strong (Danang and Quang Nam) the Red Cross actually takes some initiative beyond the project boundaries (example of providing surgery to girl with club feet). Their officials seems to be very enthusiastic as well. Perhaps part of the reason for this is that they can do the Project work at the same time as their other work - in one sense they already have a mandate to work with and for PWB. For example, many CSW's who also work for the Red Cross indicated that they conduct home \isits for the Project when out on Red Cross business. ' Hai Duong hioU was signed: 5 March 1999: lhe Danang blou is signed but not dated: the Quang Nam hloU war sign& on 21 Sovmbcr 1999. Junc ZOO; Pape 13 .-\YTP:\KD E-..:Iti.t~i~,:> i\\'TP.?.\VD \lid-Temi E\alua~ion Rcpon 4.2 Implementation Schedule Generally, the project has been implemented more slowly than was originally envisaged, as exidenced by the request for a grant extension. One reason for this is the large number of official project partners involved for whom the concepUs were novel; a second is the heavy reliance on community-level volunteers; and the third reason relates to a variety of issues inherent in the choice for working in semi-urban areas. The fist factor holds true for all development work in Vietnam; the second is unique to this project. Considerable time during the first six months of implementation was dedicated to negotiating the official Agreement documents, and formulating the official management committees (PMT and DSC) - although other work was conducted concurrently. While the PMT was formulated almost immediately in Hai Duong (Nov/98), it took one year more to fomi and finalise the District Steering Committees (NovI99). In Danang, it required several months to fomi the PMT (May/99), largely due to reluctance by DELISA to work together with the Red Cross, with the fit three DSC's finalised one month later (June199) and the fourth in August 1999. The project started in two of three provinces in October 1998: Hai Duong and Danang. At that time. discussion between WCDO and USAID resulted in the recommendation that work not be started in three locations simultaneously: it was agreed that WCDO would start in two provinces and if appropriate. phase into the third after a year. Therefore, work in Quang Nam began in October 1999. One month later, record￾breaking floods swept through Quang Nam rendering severe damage and of course hindering implementation of AVTPAWD activities: both because of issues of access and the fact that the Red Cross. primary implementing partner at the field level, was completely occupied with responding to the flooding and could not devote time or attention to this project. The first formal (institution) and informal (small business) intakes for Danang occurred in July 1999 and in August and November respectively for Hai Duong. This timing is logical considering the number of different stakeholders and the amount of training and preparation that must occur in advance. Graduation of the last AWD from informal training in Hai Duoug occurred in June 2001 and in Danang and Quang Nam a full year later (June 2002). The terminal formal training centre for Danang took place in September and for Quang Nam in November 2000, and for Hai Duong one year later (September 2001). Thus of the three project sites, just one was able to complete the full training cycle within the original timeframe. Intake for formal training was, of course, easy to plan for. Intake for informal training was conducted on an on-going basis as suitable businesses were identified. Given the desired results, the original project timeframe was too ambitious: it is not possible to accomplish the type and extent of attitudinal and social change envisioned by the project in that short period. Furthermore, it is too short a time to know definitely if the beneficiaries' employment status is sustainable. And there are elements of the original design that simply could not be implemented due to the short project timeframe. It appears that while the time stipulated as being required for actual training (6-9 mouths) was generally adequate (some businesses did indicated that it should take at least one year to train an apprentice to the highest level of skills), the project underestimated the time required for raising awareness in the community and among AWD themselves, for facilitating informal training opportunities, and for conducting job-placement. The third factor mentioned earlier, locating this project in a semi-urban location such as Quang Nam. contains some inherent difficulties such as relatively high off-fami uneniployment rates even for the non￾disabled and issues related to physical access. These factors must all be considered when implementing a community-based model such as this in future - in the Vietnam context, a minimum of three years seems a more rational timeframe for a first cycle of about 100-150 beneficiaries. I\\'TP.A\\'D hlid-Tcm E\alunrion Rcport 4.3 Project Design Overall the project components werc satisfactorily designed, although limited in scope. A more comprehensive program (ie. one including more attention to education, prevention, and treatment and the community) may have enhanced the output with minimal additional funding requirements. There are some critical elements missing in the project design and the disconnect is most clearly visible when comparing the budget against intended project activities and in rationalizing actual output (achieved satisfactorily) with intended output (too narrowly defined in Phase One). For example, the budget contains funding for provision of credit to start-up businesses for AWD's. However, the project plans do not include any reference to the 'how to' for such implementation; the timelstaff required for systems de\-elopment. responsibility for financial administration (given the multitude of partners), selection criteria, repayment mechanism, etc. Another example would be the VAC (vegetable gardens): they are mentioned in the budget esplanation as being a possible scheme that would assist AWD's in rural areas but the narrative of the proposal makes no mention of how appropriateness would be determined, etc. And no reference is made to involvement in agriculture as one positive outcome of the Project (ie. even if many of the AWD were engaged in the gardening and had improved health and community relationships as a result there is no place to reflect this against the objectives). Project planning should have included more attention to development and maintenance of an adequate information management system, including more deliberate exchange between project sites in order to foster in-house laming and speed the leaming curve. In fact, exchange visits rarely occurred and each of the Project Coordinators developed their own systems and struggled through their own issues largely without the input of their colleagues. And the evaluation was difficult to conduct in part because there was little quantifiable baseline information available against which to measure change over time. 4.4 Project Management Structure A detailed organogram included as Annex 4 gives a snapshot of the way in which progam manaement was structured. The bulk of project activities were carried out by World Concern staff and Conununity Support Workers. PMT's were not actively involved in implementation although they did receive traininss and were represented at major functions such as Disability Day. Despite plans for regular consultation between the PMT and DSC's, those meetings happened just occasionally. This is perhaps not surprising, considering that all PMT members have full-time jobs and the AWD project is just one of many responsibilities. District Steering Committees were more active in promoting awareness in their respective communities. and especially in soliciting participation from businesses initially - as well as in interacting directly with the CSWs about project issues. However, many DSC members did not - by their own account - visit the AWD families or businesses regularly. 4.4 World Concern Overall, World Concern Development Organization has done an admirable job of implementing this project which contains some unique and therefore difficult elements for the Vietnam contest. Specifically, those unique elements include a focus on adolescents with disabilities; partnering with multiple agencies; informal skills training; developing community and peer support for AWD: and emphasis on placement of students completing training. WCDO appears to have done a satisfactor). job of managing external relations. Greater attention to internal management issues would have benefited the pro,mss of Phase One. and thus achievement of goals. Given the absence of the original management personnel, it was difficult to ascertain the rationale for various internal management decisions although easy to see how some June ZOO3 Pngc I5 -\\TP:\\'VV Ei.r:uz!in:: negatively affected the Project and Project Coordinators. For instance, the nvo current Project Coordinators indicated that especially in the fmt two years of implementation, they were Frustrated at the lack of comprehensive management information provided to them by Senior Management. For example, the budget was centrally administered - Coordinators report that they were not given adequate fmancial information to be able to manage effectively. Further, Coordinators indicated that although they were given the proposal document against which to plan and manage, the proposal was not discussed in detail. Therefore they were unaware of some of the expectations for implementation - for example, the Quang Nam Project Coordinator indicated that he did not know he had the resources and mandate to recruit, train, and employ teachers for Alternative Basic Education so he did nothing to promote this during Phase One although he recognized it would have been useful. The same can be said of finances allocated to Training of Trainers - the Project Coordinators were not aware of the funds and were not aware of the expectation to do TOT. 4.4.2 Program Management Team A PMT was formed in each ProvinceDistrict, after some delay, and their involvement was adequate (although not as active as originally envisaged). Composition is not exactly as required by WCDO in relation to gender and PWD; however, GSRV ministries are represented as desired. Of I5 PMT members, 4 are female. Thus the overall ratio of men to women at the upper management level (excluding WCDO) is 61/14 or about 25% women. Perhaps more telling than gender dis￾aggregation is participation by PWD on the PMT or DSC level. Just 11175 people are PWD: of the 11, two are women. Participation of various members of PMT was not as active as anticipated, according to Project Coordinators Perhaps this should not be surprising considering their already considerable workload before taking responsibility for the Project (all are assigned by the People's Committee, none are voluntary). Meetings, especially between PMT and DSC are not occurring regularly, although relations are reported to be cordial. It is rather a one-way flow of information (reports from DSC go to the PMT). PMT meetings with WCDO or with DSC do not have official minutes although some record was kept of major issues/proceedings in Quarterly Reports. Some PMT members interviewed indicated that a few representative of their ,goup had attended at least one WCDO trainings - however, not all members had. It was not clear if opportunity had been given and they had not participated, or if there had not been any opportunity for training at all. When asked whether or not the project would be sustainable after WCDO lea\.es, two responses were very telling: one PMT director indicated that 'the model is no use if we ha\-e no budget: that authorities do not give us a budget. A project like this is not high priority." The second response was: "If WCDO finishes its work we will find more donors and that is sustainability." In other words, it seems clear that at the PMT level there is not a welldeveloped sense of ownership for this project. And there is not a good understanding about how concern for PWD might be inte-gated into other activities. The PMT did identify some strengths of the project and in general seemed appreciative of it. Strengths identified included: comprehensive nature of the project, creates opportunity for people to help themselves, it has encouraged different agencies to work together and that is unusual. Several PMTDSC members interviewed indicated that they were doubtful that the informal training model would he successful - however within the first year of implementation they could see its merits and were more supportive of it. One PMT director said bluntly: "Before the project we thought it was an impossible idea (informal training). But now we think it is possible and replicable." Weaknesses identified by the PMT representatives interviewed included: the project should focus more on awareness raising (somewhat ironic as this is one of the main responsibilities of the PMT!). the project should address a wider age-range of beneficiaries, credit/loans should be provided to AWD after completion of training to help them set up their own business, and AWD require a longer period of training than that provided by the project (this was a popular sentiment throughout the evaluation - a perception that they require at least 30 percent more time to accomplish the same learning as a 'normal person'). 4.43 District Steering Committee District Steering Committees were formed at each District, theoretically comprised of five members including representatives from DELISA, Red Cross, Department of Education and Trainii,, a some community members, and only one PWD. Of 28 DSC members in Hai Duong, eight were female. None of the 12 DCS members in Danang were female. Of the 16 DSC members in Quang Nam. two were female. In total, just 10 of the 56 DSC members were female (17%). They are involved with the Project to varying extent, depending largely on the ener3 of 1-2 active people on the Committee and most often these are Red Cross employees. The relationship between DSC and PMT seemed cordial but not consistent or very interactive. Meetings were reported as being 'occasional' and on an 'as-needed basis' but not regular. Several of the DSC indicated that they 'take orders' from the PMT and 'give reports' so it seems to be an information exchange rather than actually a partnership. Two of the DSC indicated some conhsion between their role and that of PMT (although it is clearly outlined in the MoU - an indication of poor communication behveen PMT and DSC perhaps rather than of poor planning on the prut of WCDO). The DSC members are closer to where the project activities and actual beneficiaries are, so perhaps it is not surprising that they reported interacting occasionally with beneficiary fandies. Still. they indicated that they do not play a very 'hands-on' role. Rather the major part of the work is done by the CSW's. The major source of information for WCDO about project activities and results came from CSW's through regular monthly reporting. Sometimes the DSC included additional information. Information between WCDO and DSC was primarily oral. The DSC interviewed all gave rather wide-ranging responses to the question about what their specific responsibilities were. One Danang DSC member (from Hai Chau District) indicated that his Committee's responsibility was: "cooperation among project members (PMT), conducting surveys, relating to WCDO, supervision of CSW's and group formation for AWD;, distribution of equipment. training and job placement". One Quang Nam DSC (Tien Phuoc) indicated "surveying. skills and training matching for AWD, organizing AR meetings, managing the CSW network, visit AWD. and reporting for WCDO as responsibilities of his DSC. As a minimum, such responses indicates familiarity with major project components. All four Hai Duong DSC's interviewed gave much shorter answers including: observation, communication with communities, review .AWD needs. surveying, relating to PMT, selecting AWD, monitoring beneficiaries. Most of the DSC members stated that they were not clear on their responsibility in regard to loans and equipment, which is a major services they are supposed to provide. This seems strange. given the fact that they sign agreements for each project participant which clearly spell out regulations regarding allocation of supplies and equipment and in addition WCDO circulated a document for all DSC's called "WCDO Policies of revolving equipment and start-up assistance and microcredit program". These guidelines were, according to Project Coordinators, translated into Vietnamese and distributed to DSC's after discussion and agreement on them. It would appear that PC's have been either reluctant or unable due to time or other constraints, to work with DSC's to ensure that there is a solid system of accountability for equipment provided through the project. It is true that one DSC interviewed, and two of the business owners interviewed in Quang Nam indicated that they had June ZOO3 Pogc 17 \\TP-\\\D E~sitilirioi: AVTPAWD Mid-Tcnn Euluation Rcpm experience with returning equipment to the DSC after an AWD had left employment. No problems or difficulties were reported. DSC members interviewed all indicated that their Committees were not very pro-active in promoting awareness relating to disabilities, although all of them participate actively in Disability Day (April) and in occasional media features (television or radio) that highlight the project. Their role in Awareness Raising then, seems rather passive, given that directives for Disability Day come from central authorities. It is worth noting that DSC members indicated the major strength of the project as relating to changes in perception and attitude and behavior and social integration, rather than in issues of employment or income. 4.4.4 Community Support Workers During Phase One, there were a total of 52 CSW's, 31 women and 21 men. They numbered 2-6 per district depending upon area covered and target population. In theory, the CSW's are supposed to be voluntary members of the community-at-large. In practice. the vast majority of them are Red Cross Volunteers (which is a paid position) and the majority of them have other employment as well, most often government employment (ie. member of the Women's Union, etc.). In Danang and Quang Nam, in contrast with Hai Duong, a much larger proportion are actually Red Cross Volunteers than simply government employees assigned to the task. The difference in attitude and work was very evident to the evaluation team, although impossible to quantify. For selection, WCDO verbally communicated its criteria to the DSC and PMT (used to community work and like it; good hearts; women preferred) and asked for them to provide candidates. For the start in Phase One, the candidates were then sent to training and the final CSW's were selected based on observation of them during the training. In the event of a CSW leaving their posit~on. WCDO works with the DSC to select another person. Although technically it is the DSC who selects CSW's, WCDO Project Coordinators all reported that they feel they have sufficient influence on who is selected and that they can also influence to remove and replace ineffective CSW's. Although the CSW role is technically a voluntary one, the project did provide various incentives to CSW's, including: Transport Allowance (150,000 VND or USD SlOlmonth for up to 10 AWD and 15,000 dong per child supervised per month for a number over 10 --to cover transport. food while in the field, and incidentals); Income Support (a loan of up to 2.5 million VND - S166 USD - for starting up a personal business, repayable to the DSC); and provision of a bicycle upon request (-gant basis). One motorcycle was provided by the project to one DSC (in Quang Nam, Tien Phuoc District). In both Danang and Quang Nam, approximately 90 percent of CSWs availed themselves of the Income supports and transport (grant of one bicycle). In Hai Duong, none of the CSW's took advantage of either benefit. This is because as government employees they already have motorcycles and they all felt the loan amount to be too small to bother with. It is difficult to estimate the influence of incentives on the decision of individual CSWs to take on this responsibility. Not one CSW mentioned these incentives as rationale for taking up the post: the two major reasons for becoming a CSW included being asked by a member of the DSC and feeling compelled to accept; and a desire to help people who are in difficult circumstances. 411 CSW's who took out the loan reponed that it was very useful for improving their ow family income. and nac mon oRm used b? the - to sel up a small busin- such as selling ice cream or vegnable: or. in Quang Nam, purchase to wnle. June 2003 Page IS .A\'TP.iU I> E:.hxi,w r\\'TP4WD Mid-Tern, Evalueion Renon The CSW's are most closely aligned with Project philosophy when comparing CSW, DSC. and PMT. They generally defmed success in terms of life improvement and social inte-gation for the AWD, rather than in purely economic terms. They reported having applied information from trainings to work and life, with 'communication' cited as the most useful topic. Inteniew results indicated that CSW's recognized that both 'head' and 'heart' knowledge are important when working with AWD. It is difficult to qualify the performance results of CSW's, partly because there was no formal job description provided for them and thus no official standard against which to assess perfomlance. However, the CSW's and Project Coordinators all reported having an excellent working relationship and in the case of Danang and Quang Nam, personal relationshipls with each other. The relationship between WCDO and CSW's in Hai Duong was different and more difficult, attributable according to the current PC, to the fact that CSW's are all appointed and have government positions so the CSW job is merely an add-on to existing responsibilities. In all three sites, WCDO indicated that reporting from CSW's was thorough and accurate (the presence of basic reporting forms was verified by the evaluation team in all three provinces) and that communication was very frequent. Healthy interaction between CSW's and WCDO staff was observed by the evaluation team. 4.5 Project Beneficiary Selection 45.1 Selection Process for A WI) The number of actual project beneficiaries slightly exceeded the number specified in the original plan: 519 rather than 500 (233 in Hai Duong, 133 in Danang, and 153 in Quang Nam). Of the 519. 257 (49 percent were male) and 262 (51 percent were female). There was no appreciable difference in results (ie. completion of training, employment rate) between males and females. The selection process, as outlined by the Danang PC was as follows: a) Introduce project concept to DSC - and obtain assistance in locating and listing AWD within their area ofjurisdiction. b) Train in methodology of survey for DSC, Red Cross volunteers, and CSW's. c) Conduct house-to-house BSmA. d) Analyse survey results and do a pre-selection of eligible candidates (it was obvious that some children were too severely disabled to participate or had other circumstances that would not allow them into the project). e) Giving preference to the oldest AWD on the pre-selected list (because of the project's defmition of adolescent), return to homes and verify information. f) Submit the smaller list of potential beneficiaries to the DSC for final approval. g) Return to families and inform them that their child would be enrolled in the project if they wished. h) Sign a training agreement with both the family and the business. In this a-geement. WCDO agreed to pay training fees for up to nine months. 4.5.2 Selection of informal businesses During Phase One, 64 percent of total AWD were trained by informal businesses plus an additional 6 percent had to undergo re-training in an informal business after completing formal training. The selection of businesses occurred gradually and almost simultaneously with selection of AWD (ie. throughout Phase One there was on-going intake of businesses, as the number of AWD -mw) as great effort was made to match the desires of the AWD and the businesses available. In other xords. businesses were selected not simply because they wanted to be in the project, but also because of: Junc 2003 Page 19 :\\TP:\KD E\ldux/oi? Reputation and stability (29 of the 35 businesses interviewed reported being in business ten years or longer), which correlates with employment prospects. Stability and size (8135 interviewed had ten or more employees; 6 had 5-9 employees). = Proximity to the AWD - the closer, the easier to attend training regularly and then to gain employment after training. Capacitylwillingness to train someone with a particular type of disability. Informal training oppomnities were identified by DSC, CSW's and by WCW staff. OOAen there was a relational connection between the business and the AWD or the person who identified the shop. Rather than being seen as a negative, this personal connection actually appears to ha% been a strength of the program - business owners were probably less likely to quit or give up on a diFficult AWD because they felt a sense of responsibility and obligation to make the relationship and training succeed. 4.5.3 CEDC The total number of children in especially difficult circumstances (CEDC) participating in this project was 40, or just 8 percent of the total number of beneficiaries. The project a,geernent allowed for up to 20 percent of total beneficiaries to be comprised of CEDC. The CEDC were selected at the same time as AWD were, and identified in the course of the community survey. All CEDC on the Program graduated successfully for their training. CEDC were not specifically reviewed by the evaluation team. 4.5.4 Types of disabiliw AWD with mobility impairment comprised nearly half of the total beneficiary population (47%). Ln order of prevalence the other types of disabilities serviced in this project are: hearing and speaking impaired, visual impaired, speaking impaired, learning impaired, and hearing impaired. It would be instructive to know the types and proportion of disabilities within the WCDO population that are preventable. Such information would be useful in designing a prevention conpnent to future disability projectls - and in the immediate term, could he used to educate communities in prevention. treatment, and care of conditions that lead to long-term disability Total AWD by Disabiliw Type AVTPAWD Mid-Tem, E\aiualiun Rupon 4.6 Baseline Situational Analysis (BSA) A baseline situational analysis (BSA)was conducted at each of the three provinces early in the project. Survey implementation seems to have been a very consultative affair involving all major stakeholders and as such likely proved valuable in engendering feelings of ownership in participants. The BSA contained a large number of questions aimed at determining whether or not the AWD in question would be eligible for participation in the project: what is the general situation of the family, how is the health of the AWD, did shehe have adequate mobility, employment history, AWD's education level, what kinds of activities does shehe prefer, what kind of job shehe might be interested in doing, and so forth. Probably due to time constraints only a small portion of the 8-page questionnaire was analyzed -just enough to prokide WCDO staff with an idea as to whether or not a particular AWD could participate and would thus wanant a follow￾up visit. Data from districts where WCDO decided not to work was not even analyzed (this was the case in 3 districts of Quang Nam). It is unfortunate that this valuable information was not thoroughly analyzed and recorded9- it would have provided solid baseline data against which to assess attitudinal, employment, health, and quality of life change over time. There is also no indication that it was used to influence site location, beneficiary numbers. or any other aspect of programming except the selection of AWD. In hture, World Concern should take more care to utilize the information that it collects on beneficiary populatiods - or to take a more basic approach to collection of information 4.7 Training Needs Assessment (TNA) The 'training needs assessment' (TNA) included in this project was comprised of two parts: mA of the AWD (conducted as part of the BSA - a full three pages of the BSA was dedicated to this tyw of information) and a TNA for businesses and institutions which might be eligible for in-take and training/employment of AWD. The 3-page form used seems to have provided an excellent reference for discussion and business owners interviewed during the evaluation all felt that they had been adequately consulted on participation in the project. Again, however, there was no visible effort made by WCW to store and analyze this data for future reference and monitoring. 4.8 Participation and Monitoring Not surprisingly, the degree and level of participation varied between the major stakeholder groups. PMT's were least involved overall; then DSC; then CSW's. As a minimum form of participation, most of major stakeholders interviewed (with the exception of parents and AWD) indicated that they had participated in at least one formal training conducted by WCDO. Many project activities are more accurately characterized as consultative rather than truly participatory. For instance, the entire TNAJBSA was developed by three WCDO staff members and then stakeholders were trained in how to administer the questiomairels and utilize the information. Beneficiaries were not involved in the development of forms or determining the process - this is not to say it was not effective, but merely to distinguish between a 'participatory' and 'consultative' approach to development pro-mming. In fact. the TNA and BSA were reported to have been very successfid at involving stakeholders and beneficiaries: Businesses reported 100 percent that they had been adequately consulted about the placement of AWD within their shop and most AWD indicated that they had also been adequately consulted. Still, there was significant scope for greater involvement of the beneficiaries in planning, goal setting, identifying indicators, and tracking results. Also, WCDO could have done more to utilize the AWD themselves as promoters of awareness in their communities rather than relying almost solely on the DSC and CSW's. 'At the time of the evaluation, a large box containing the original survey farms for Quang Nam and Danang was bmught into he oflice fmm r storexom; only a fraction of the farms completed for Hai Duong were available at the Hanoi office (the cum1 PC indicated hey may hare kpn hw in an office move). The servicedelivery approach meant that the WCDO staff were quite involved in peer groups. Eventually responsibility devolved to CSW's. Not until near the end of Phase One was more attention given to enabling and encouraging AWD to run their own meetings. Many were still running during the time of the evaluation. but there is still a significant degree of WCDO involvement so it is difficult to know whether or not the groups would continue in the absence of any WCDO involvement. Likely they will, given the enthusiastic response of most AWD when asked about the benefits of the peer groupls. At a minimum. individual relationships were reported to likely continue even if the formal peer groupls stop meeting. The Red Cross seemed to be a very able partner organization. Most monitoring was conducted by WCDO staff themselves as well as by the CSW's. It was to be one of the a functions of the DSC but that largely did not materialise. The DSC indicated that they monitored primarily by discussing with CSWs the monthly report prepared by CSW's. Infrequently, a DSC member would visit an AWD family or participating business and especially if there were difficulties encountered by the CSW. Monitoring and reporting were largely informal -"we work in the same office." WCDO quarterly reports were not a very effective progress monitoring tool; although records regularly compiled by the Project Coordinators would have provided a useful monitoring tool if the records had been consulted and systematically organised. 4.9 Advocacy The current policy environment in Vietnam seems to he very conducive to PWD, though of course practical application of policy is another issues. For instance, all of the four formal institutions interviewed reported that they have a centrally-mandated quota of PWD that they must train and find employment for. Vocational training is tax free; everyone who is disabled, poor, orphaned, or a relocated student can study at a government vocational center free of charge. Danang City reported giving a subsidy to vocational training centers for each PWD who studied there. Vocational training centers which have at least 70 percent disabled students are entitled to other types of tax exemption and are eligible for loans from the PWD job fund. A second example is the policy "Decision 81lCP dated 2311 111995" which stipulates that there is tau exemption in income tax, benefit tax, house tax, and land tax for "PWD businesses" (defmed as ones who have at least 10 disabled employees, and 51 percent of total employees are disabled). The same "Decision" document states that if businesses do not have the stipulated number of PWD" then they must make a monthly contribution to a PWD Job Fund (contribution is calculated by the shortfall of PWD x minimum salary of a normal worker): Given this positive environment, and the difference between policy and practice, it would be usehl for WCDO to seek strategic ways to influence the outworking of such policies if they would benefit project participants. One starting point would be to educate participants and beneficiaries in the legal framework governing work and employment and training of PWD. One DSC representative interviewed indicated that the lack of advocacy on WCDO's part was a weakness in the Project. He was challenged to consider his own agency's role in advocating on behalf of the disabled as perhaps being more appropriate within the Vietnamese context. '' Businesses involved in pmduction. electicity, metallurgy, chemical, pemleum. mining. aqwsulture conaruclion. ~alia, have to hire at least 2 percent PWD: omer types of business must hire at least 3 percent AWD (of total \vorkfarce). .A\'TPA\\'D )lid-Tcmi E\alun~ion Rcpon 5.0 PROJECT OUTCOMES Project outcomes were reviewed and assessed in terms of the four major components set out in the objectives (Awareness Raising, Vocational Training, Income and Employment, Support Nehvorks). In general, planned outcomes were achieved and positive changes reported at least some of all the stakeholder groups interviewed. However, it was very difficult to quantify changes given the almost total absence of any baseline data against which to measure. Also, there were many project outputs and outcomes not anticipated in the original design, and it was difficult to capture those as the evaluation team was constrained to assess progress against original goals and objectives. 5.1 Awareness Raising 5.1.1 World Concern Relief and Development Organization Since 1999 World Concern has produced a variety of awareness-raising materials including an annual calendar, a laminated "Flipchart For Disability Awareness" featuring sketches by an .4WD in Danang, as well as various leaflets and brochures in English and Vietnamese languages. The flipchart was visible in most of the Red Cross offices visited during the evaluation. During the project WCDO was creative in organizing major events to raise the profile of PWD within their communities. For instance, in 2000 around the Day of the Disabled (18 April). WCDO organized a Drawing and Writing Contest for students of normal schools, special schools, centers and villages where PWD live. Hundreds of people participated in the contest and obse~ed the exhibition, which was open for six days. No specific effort was made to assess impact. In addition, during the life of the project, district radio stations and occasionally television featured the project and activities especially around the Day of the Disabled. The extent of media coverage was noted by staff in quarterly reports, but not systematically recorded or collected in the office. 5.1.2 Program Management Team (PMV According to the interviewls, at least one member of each PMT participated in formal AR training by WCDO. It would seem from interviews, however, that awareness was raised as much by the process of actual project implementation as formal trainings. The Danang PMT indicated: "Our axvareness increased as we worked through the Project - after the Project we were interested in things like social integration." None of the PMT indicated that they initiated any AR acti~ities themselves - although all reported that they were very involved in activities around the annual Disability Day (IS April). Note: their involvement would occur regardless of WCDO's project because they are part of DELISA, the government agency tasked with looking after PWD. The Danang PMT indicated also that they were surprised about the ability of PWD to run and be involved in Peer Groups. This had influence on their planning to start-up Youth Disability Associations in their Province. 5.1.3 District Steering Committee (DSC) As with the PMT, DSC members also reported inconsistent attendance at formal AR trainings conducted by WCDO. And those who had attended trainings were very vague about the things they had actually learned, perhaps because of the time which has elapsed behveen then and now more than for any other reason. Quang Nam DSC members more consistently reported having attended WCDO training events. When asked to cite evidence of change since the AR training. they responded with the following: A\'TP-\\VD !.lid-Tmn E\.,~luarion Rcpon = We learned some new things like how to appropriately refer to disabled people. We understand now that the PWD can do things for themselves. My learning helped me to know how to convince businesses to get over their doubts about training AWD. All DSC members from both Quang Nam and Danang Provinces indicated that they do not enzage in AR activities of their own initiative (they do report participating in the annual Disabled day events). Two of three Hai Duong DSC representatives interviewed all indicated that their Conunittees do AR activities such as "giving relief supplies and subsidies" and "giving gifts on special days." Again, it was clear from interviews that DSC members equate Awareness Raising with all types of interaction they have with the AWD, AWD families, and businesses. This would appear to be a reflection of the vagueness of WCDO itself concerning "Awareness Raising'.. The entire educational effort would benefit from strategic thinking that would contribute to development of a very few basic and key ideas and messages which can easily be promoted and then change in knowledge (and perhaps attitude) would also likely be more easily monitored. 5.1.4 Community Support Workers (CSWj All CSWs undenvent AR training. The majority interviewed indicated that they see AR as one of their primary responsibilities and are active in doing this. CSW's on average indicated that they visit their target AWD and family at least once per month if not more frequently (the reported Frequency of visits was slightly less in Hai Duong than in either Danang or Quang Nam). .Almost all CSW's equated visits to families and businesses for monitoring, with Awareness Raising: they defined their AR as talking to people within their sphere of influence such as families of AWD's. businesses. neighbors, customers, and the AWD themselves. They also indicated that they are involved in organizing special events such as the peer group meetings, outings for the AWD, and annual camp. Most CSW's (80 percent of those interviewed) gave examples of how AR training had affected then1 personally with such responses as: = Improved my capacity to be encouraging of the AWD for their small improvements. Increased my sympathy and capacity for appropriate interaction. Before training I found it difficult to relate to AWD but this became easier after AR training. = Now I understand psychology of AWD and their problems and feelings. Know better how to communicate with AWD. = I know more about disability issues after the training. CSW's reported that they had given Awareness Raising training to their target families: SO-100 percent. Further, almost all CSW's interviewed provided examples of how training and involven~ent in the Project had changed behavior among their target population. = Increased confidence of families to share their feelings and opinions, The parents treat their children differently now - they want to make their (disabled) kids more independent. The AWD don't feel so isolated or lonely now. = One told the story of a mother who didn't treat her disabled daughter well. After spending time with the CSW who told her how to treat the child appropriately, it as reported by the AWD to the CSW that the mother had stopped beating her. Junc 2003 Page 24 ~-\VTP:\\VD E\3!c:::i,:;t Another family said they doubted the ability of their poliodisabled daughter to learn skills. Now she has become a successful hairdresser and they talk to their neighbors about her achievement! = In another instance, the CSW and a mother were not confident in the capacity of the AWD son to learn; but now the son is 'a good woodcarver'. Families encourage their AWD more; they give compliments more Frequently. = Originally people in the community were skeptical about this Project - but now they are understanding and some have helped us to find businesses in which to place AWD in the future. 5.1.5 A WD and Families A full 86 percent of AWD interviewed said that the Project had helped to raise awareness about disabilities within their own families: most frequently mentioned were increases in caring, more understanding, sympathy, help, and better communication. As for self-awareness, when asked to compare their awareness of seven issues beford after the project AWD indicated the following: No AWD indicated that there was no change in understanding on the issues. - 8% indicated change of an average of half-increment in knowledge about the issues. 18% indicated change of 1 increment in knowledge. = 21% indicated change of 1.5. * 31% indicated change of 2 increments. = 9% indicated change of 2.5 increments. - 8% indicated change of 3 increments. As for the AWD families, more than 80 percent indicated that the Project had helped to raise awareness about issues related to disabilities. Using the same scale as that for the AWD. families rated their beforelafter knowledge as follows: 12% indicated that there was no change in understanding on the issues. 10% indicated change of an average of a half-increment (5) in knowledge about the issues. = 21% indicated change of 1 increment in 'knowledge. = 15% indicated change of 1.5. 22% indicated change of 2 increments. = 6% indicated change of 2.5 increments. 5% indicated change of 3 increments. When asked if the Project had made a change in their own behavior and response to their disabled children, 16 percent of families said there had been no change. The remainder (84 percent) said there had been changes and included such expressions as more: encouragement, sympathy, care. and understanding toward their children. 5.1.6 Business and Formal Training Institutes Representatives of participating formal training institutes were vague in their response to the question of whether or not the Project improved their awareness of issues related to disabilities and disabled people. It is unlikely that the Project would have contributed more than the influence they already receive from DELISA and perhaps the government influence would 'innoculate' them to similar (or related but perhaps not similar!) messages from other sources. Junc 2003 Page 25 :\VTP:\\VD E.\.:iuxi+:t As for private or informal business owners, just one indicated there had been no change in his awareness: 21 indicated there had been change of +I increment and 15 indicated there had been increased awareness of +2 increments on the scale presented during the interview. \%en asked whether the change in awareness as due the Project, business owners responded by indicating four stimulants of change: exposure to AWD and the actual experience of training them (de-mystiIj~g disability): attending group meetings with their AWD; = visits from WCDO staff and Red Cross volunteers; and the least number stated that formal AR training from WCDO helped to improve their awareness". All business owners interviewed indicated that they regarded the project as a success: success was most often stated in terms of being able to earn income, but some of the owners also said it was successful because the AWD had learned skills, had employment, had learned some life-skills (was able to communicate with people and society outside their homels), had improved social skills and communication skills and confidence. These responses indicate a broad understanding of issues related to AWD and PWD. The majority of informal training sites indicated that they would continue to hire AIVD in the future, even if WCDO was not involved in the process. However, in most instances the business owner made it clear that they would not initiate training of AWD - they would have to be approached. 5.2 Adaptive Vocational Training 5.2.1 Training Selection and Placement A total of 190 AWD were trained in institutes or formal centers; 339 were trained in informal businesses or had a combination of formal center and informal business training. Training occurred in a wide variety of skills. The most popular skill choice was sewing (46%), then electrical repair (12%), embroidery (1 I%), motorcycle repair (8%), bicycle repair and hairdressing (4% each), and then several other skills such as woodworking, toothpick making, watch repair, shoe making, and so forth. AWD Types of Training Skills " In facl, mare than half the businesses intewiewed indicated that they had newr attended a formal AR mining by WCDO - this is m ~prisimg as businesses were inducted into the Project on an on-going basis and the AR trainings w~c major undmakinps and thus not rq- frequently conducted. Decisions about training type and placement were made primarily by DSC representatives. WCDO staff, CSW, the AWD, and AWD families as a result of the Training Needs .Assessment and the Baseline Survey Assessment. Also, 100 percent of informal business trainers interviewed indicated that they had adequately been consulted on the placement process and AWD placed within their shop. Overall, there appears to have been a good 'fit' between AWD and placement althou~h 22 percent of AWD interviewed reported that skills training did not meet their expectations, income level demands, disability, or need for stimulation (boring). Training and Employment Skills Comparison I3 Employment 5.2.2 Equipment Support Equipment support in the form of sewing machines, cabinets, tables, drills, etc. was proxided to all four of the formal centers where WCDO placed AWD through this project (Trade Union Centre and Social Protection Centre, Hai Duong; Job Service Centre, Quang Nam; Employment Senices Center, Danang). Decisions for the specific types of equipment were made by the head of each centre in conjunction with World Concern project staff and then WCDO took responsibility for actually procuring the machinery and equipment (following the standard system of obtaining quotations from at least three different suppliers). The evaluation team obsen-ed some of the donated items during their visit to each centre and all appeared to be in good working condition. The quality of machinery was assessed by the centre directorls as being adequate, although not top quality. Actual expenditure on this budget item was only about 70 percent of the amount allocated. June 2003 Page 27 ..\VTP:\\YD E\;h::tim r\VTP.'\\VD \lid-Tcnn E\aluntion Rcpon Equipment for informal training businesses is covered in the section on revolving equipment and supplies. 5.2.3 Site Zmprovernent Site improvement was planned and executed for the formal training centres, consisting primarily of modification to bathrooms (widened doonvays, handrails), dormitory construction anrYor improvement (beds, fans), and installation of ramps. AWD were not consulted on these building changes, according to the WCDO staff and centre management. The evaluation team observed any of these building changes during visits to the sites. No site improvement was conducted in any of the informal businesses, although this appears to have been part of the original project plan. In Quang Nam, WCDO staff indicated that they had assessed the need for such changes during the Training Needs Assessment and purposefully selected businesses that did not require improvements. WCDO staff at other sites did not echo this care in the selection process. Furthermore, in some business locations, the evaluation team observed that there could have been improvements made to working conditions in general (very poor toilet facilities. poor lighting, unhygienic kitchen facilities, etc.) - in those cases, perhaps a cost-sharing arrangement with the owner would have provided a good solution". During the survey, the majority of business owners indicated that they did not see a need for structural improvements either to their own shops or to public facilities - this may simply have been a result of lack of exposure to the possibilities or to their limited exposure to types of disabilities (for instance, if a shop owner had a deaf adolescent in training, shehe may conclude that structural improvements are not necessary). There was considerable underexpenditure against this budget line item (60% or only S21.000 out of the budgeted $34,000). Given the fact that the project recognised this need, management should either have been more pro-active to allocate hds for improvements to informal businesses or perhaps considered allocating it to homes of AWD or to other public buildings such as schools which do, or will in the future, service AWD. For home improvements the cost-benefit ratio would be significantly less than that for changes to formal centres or public facilities, and this would ha= the added value of raising community awareness about the need for improvements within the PWD environment. 5.2.4 Trainers - Mobile and PWD The original plan, according to the budget, was to hire mobile trainerls who would better serve AWD who perhaps were limited in their own movement or had limited opportunity to learn given their physical location. However, it proved difficult to find appropriate people who were willing to do this and so the idea was not pursued. There was also some reference to hiring and training PWD to continue as trainers in the Project as they could better related to the .4WD and also \vould sene as a model to the communities and to the AWD - again this idea was not conscientiously pursued during Phase One due primarily to limited time (as reported by Project Coordinatoris in Danang and Quang Nam). " In fact the original proposal makes provision for msuring that wininglemploymm1 sire are safe and healthy but this was m m ab\i~t~ pan ofthe Phase One. June ZOO3 Pagu ZS .A\.?P..\\'V9 EL.~u:%~+:> AVTPAWD Mid-Tcnn Euluation Rcpon 5.2.5 Alternative Basic Education (ABE) The formal education achievement levels of AWD are considerably lower than for the general population. Half of the AWD in project have only attended primary school (6 percent of the AWD reported having no formal education whatsoever and 44 percent had from 1-5 years). Just 5 percent reported attending upper secondary school (level 10-12) and 45 percent reported attending at least one level of lower secondary scbool (level 6-9). For this reason, World Concern's project plan included provision of so-called Alternative Basic Education (ABE). Essentially, this was meant to be an effort to ensure basic literacy skills (defined as completion of level 1-3 of standard school curriculum). Therefore, one would have espected about half of the beneficiaries to need ABE. In actuality, this activity did not occur as originally planned. Formal involvement in provision of ABE only took place in Hai Duong, through ABE instruction for 35 students. WCDO recruiied and hired a teacher in each of the districts - it was an oversight that these teachers did not receive any type of training or awareness raising from World Concern. Many businesses expressed a desire for the AWD placed in their training care to have had at least basic literacy, and particularly mathematical calculation skills. 5.2.6 Graduation Results The WCDO Project Records indicate that just four students who slatted training did not complete their full course, either formal or informal. 5.3 Employment and Income Generation 5.3.1 Record of Employmeni According to data provided by Project Coordinators, and compiled from reports submitted by the CSW's, a total of 86 percent of AWD trained in Phase One were employed as of March 2003'" Of those employed, 74 percent were employed within their home province and 12 percent were employed outside their province. In total, 74 AWD (about 75 percent of those employed) indicated that they had had continuous employment since graduation, 22 did not have continuous employment (about 21 percent). and four were registered as unknown (less than 5 percent). Interestingly, although they indicated willingness to continue participating in training and employment of AWD, business owners indicated that one of their frustrations was that the AWD do not learn as quickly as a 'normal person' and that they made more mistakes. One of the reasons cited for this was that the disabled are often not well socialized so they do not know how to express their own frustration or how to ask questions; they are usually less formally educated; and they may lack some basic motor-skills required for certain tasks either because of their disability or simply through lack of stimulation. A second frustration was the 'low productivity of AWD' (businesses estimated that productitity of an AWD was about 30 percent less on average than that of a 'normal person'). It would be useful to conduct some applied research to address both of these questions - what is the objectively verifiable " Of he 14 percent unemployed. 9 pecenl were unemployed for unknown rearon's and 5 percent wen$ qistRcd as 'riclt' and thmfarr ma employable. June 2003 Page 29 :\\'l'P=\XD E\~:!mtiwt situation? Is it merely another incorrect perception, which perpetuates negative and untrue stereotypes? 5.3.2 Income and Lifestyle During the interviews, 67 percent of AWD indicated that they were currently earning an income (a figure significantly lower than the comprehensive project records indicate - the lower results From the evaluation could be due to sampling). The lowest level of income was recorded at S3.5. Only four interviewed indicated they were earning more than S35lmonth. S16 was the reported median income. From the records the lowest level of income was recorded as $3.5 per month. The highest levels were Quang Nam at $45, Da Nang at $55, and Hai Duong at $65 per month. Median for Quang Nam was $13.5 while the other two provinces recorded $20 per montb. The upper level of income rose about $5 for all provinces from the start of employment until the present. Details Questions were asked during the evaluation whether families though that their AiVD child's income contributed to an increase in overall family living standards. About 65 percent of families said that there bad been at least some improvement in their standard of living due to their child's participation in the Project. However, when asked to rank the amount of change in their lifestyle over the project period time, the same proportion (67 percent) said there was no change. The remaining 33 percent said there had been a change only by one increment (on a scale of 0-3). From the interviews the AWD indicated that their families had voiced expectations about getting good training and a satisfying job after the Project; few indicated self-sufficiency as a goal or expectation. This being the case, the Project can be considered successful in the eyes of primary beneficiaries almost regardless of the income they achieve. 5.3.3 Small-scale credit Small-scale credit was not promoted or implemented by the project - it was considered too difficult to manage given the complexity of the existing activities. And Project Coordinators seriously doubted the capacity of the AWD to repay the loans. Furthemore, the project documentation did nor include any sort of plans for utilization of this budget line-item. No allowance was made for the time and training and technical expertise that would have been required to actually implement an appropriate and successful credit scheme. More than half of the stakeholders interviewed during the e\:aluation indicated that the lack of small￾scale credit with which to help AWD start businesses was a weakness in the Project. Still, the high employment rate at project end is testament to the ability of trained AWD to gain employment without input of financial credit. And some AWD who did set up their own businesses in fact received in-kind credit through the provision of the machineslsupplies they had used during their training period. 5.3.4 Revolving equipment and supplies support From the budget information, it is clear that this was meant to include money for setting up small businesses, starting gardens, mobility assistance, and so forth. All of the businesses inteniewed during the evaluation responded affirmatively to the question about whether or not they had received equipment andlor material support from the Projecti4. However, there was confusion as to the status "The Project sits all kept delailed records on the type of equipment. lolal eosr. dale of receipt. and cumnt status (including hkm. Lwmved by AWD, rerumed to Red Cross, etc.). Junc 2003 Page 50 .AVTP.4KD E:A:xion of the equipment - was it a loan, grant, or 'borrow'? Each business owner did sign a contract outlining their responsibilities: the training contract had four signatories which included the AWDI WCDO, business owner, and DSC. A second document governing use of equipment and supplies was one called "WCDO Policy of revolving equipment and start-up assistance and micro-credit program." It would appear that this policy was more of a guideline for WCDO staff in discussion with the DSC about their role, than with the business owners. WCDO left it to DSC to explain the system for use and return of equipment and this discussion apparently did not happen consistently. In practice however, it seems to be working. A few businesses and a few DSC indicated that they had in fact taken hack some equipment given to a business after the AWD left the business and that this had been done with little difficulty. Furthermore, several AWD inteniewed indicated habing received their equipment from the project - they simply took it with them from the place where they had been trained to use in setting up their own shop. No gardens (VAC) were started as a result of this project. The concept contained in the original proposal was apparently meant to apply to rural areas where the only option for emplo~ment might be some form of agriculture. However it was not a welldeveloped notion in the proposal and does not appear to have been a concept circulated among the Project team. In Quang Nm, the most rural of the three project sites, when asked why no loans were given for agiculture, the Project Coordinator indicated that this would have been beneath the dignity of the AWD who go for skills training - skills and employment was expected to be 'off farm'. Many beneficiaries reported to have received mobility assistance (wheelchairs, bikes. walking aids. etc.) although no exact figures were obtained during the eva~uation'~. In the original proposal it appears that this was to have been granted on a borrowthen purchase basis but in fact mobility assistance was provided free of charge to AWD as required and appropriate to their disability. Supportive Networks Peer groups comprised of AWD enrolled in the Project are the major type of Support Nehvork for AWD developed through Phase One activities. Groups met on a monthly basis, and the number of AWD involved depended almost entirely on location. The groups tended to be smaller but more numerous in Quang Nam where beneficiaries are more geographically spread out. When asked what activities were conducted during the Group meetings, CSW's indicated five major things: sharing experiences, singing and games, making friends, discussing current issues (such as smoking, AIDS, drugs, prostitution), and teaching lifeskills (health, social skills, making reputable friends, boylgirl issues). When asked about activities and benefits of the Peer Groups, the AWD themselves answered that they participated for social benefits (company, making friends), sharing experiences, they liked the music and games, and group activities which helped in making them more confident, improving social skills, fun, happiness, and knowledge about personal health. In addition to monthly Peer Group meetings, occasionally in each province a group of Groups meets together; and occasionally excursions were also organised. The most often mentioned excursion was the annual camping trip. The majority of AWD responding to the survey indicated that this as a real highlight and fun and special time for them. 'I Records are available at the site oftices. June 200: Pagc 3 1 .A\'TP-\\\I) E\ain>tio:i u# rid W In addition to the April 18 activities highlighting disabilities, each September World Concern is involved with DELISA in making the mid-autumn festival a special time for AWD - for instance in1999 WCDO collaborated with the Job Service CentreISocial Protection Department in Hai Duong to host a party for more than 200 disabled children. 5.4.2 Lifskilk Training Life-skills training was intended to be part of the curriculum for Peer Groups. In practice. it did not occur in a very systematic or structured way across all three project sites. Only in Hai Duon, 0 was there formal training for the CSW's - a 2day workshop conducted by the two social work interns. There was no follow-up to the workshop, although the CSW's were given some written materials they could follow for themselves. In Danang, there was one 2day workshop on lifeskills (using the same trainers, materials, and format as in Hai Duong) conducted for 40 A\VD. Otherwise. such training occurred informally in the groups, based on specific needs (ie. if one A\\?> presented with an eye infection the group would discuss prevention, treatment, and care), and using infom~ation as CSW's of the Project Coordinator took personal initiatixre to obtain it. Quang Nam Project Coordinator indicated that because they were already behind schedule they focused solely on group formation during year One and only started to address lifeskills in a rather informal way - that is, the PC talked to CSW's about some basic health and con~munications issues that would be important for AWD's and the CSW's took this to the Group. This area of lifeskills training is reportedly being addressed during Phase Two. About 83 percent of families interviewed indicated that their child had been taught some life skills in the Peer Group meetings and that generally the skills relating to attitude and social interaction as well as to personal health and hygiene were useful. The comparison of parents reporting and self￾reporting by AWD on changes is as follows: General comments made by parents included that their children were more confident after participation in the Group, there was better health and happiness, the children were more independent, had better attitudes, and had more friends. In addition, they noticed that their children were more open in communication, more understanding, considerate, polite, and more knowledgeable. General comments by the AWD on what they experienced as benefits included: more confident* better behaved, better communication, fit into society better, enjoy life more. Given the positive changes reported and the apparent felt-need for training in life skills, WCDO should in future be more systematic and methodical about this aspect of the project: assessing learning needs in a participatory manner, defining learning goals, and developing an appropriate cuniculum as well as providing training for the CSW's in how to utilize the educational materials. 5.5 Formal and Informal Training Approaches 5.5.1 Formal vs. the Informal Approach During Phase One, a relatively large proportion of students were placed in formal training (190 A\VD or 36 percent of overall trainingl6). This was done primarily because project partners (PMT) expressed some misgivings about the possible success of informal apprenticeship and training arrangements and even WCDO Project staff were dubious about the potential success of informal training arrangements. Interestingly. although the use of 'informal business' as a place for training adolescents with disabilities for employment was unique in Vietnam and certainly a strange concept for the PMT and People's Committee. In fact the general concept of an 'apprenticeship' is not uncommon in Vietnam. Many business owners participating in this project indicated that they were already doing this with 'normal' adolescents and youth before the Project - as a way of generating more income for themselves and of passing on their skills. Usually parents approach a shop and negotiate with the owner for a training period and fee and the relationship be-&s. In many cases, it was reported by small business owners that AWD who had successfully completed a formal training course had to be 're-trained' (for 1-3 months) when they were placed in a small business. The cause for this is not immediately clear. From the interviews there was a suggestion of several reasons: a reflection on the quality of instruction in formal training institutes; a reflection of the predominantly theoretical nature of instruction at the formal institutes (little chance for practical application); a result of poor retention due to the AWD's relatively low education level; learning difficulties exacerbated by insufficient one-on-one instruction from a busy teacher with a large class; timidity on the part of the AWD who must at the same time as starting work, become familiar with the shop owner and hisher way of production and of doing business 5.5.2 Relationship between Training Type and Employnrent Outconres One of the purposes of the evaluation, as is stated in the Terms of Reference, was to: " ..... assess whether the model used for raising awareness, vocational training and employment and income generation, and social integration and support for A\VD is valid for the semi-rural and rural environments of Vietnam. Particular focus will be upon the model for vocational training and employment especially in the non-formal sector". An analysis of the data from records kept by the Project Coordinators was undertaken to compare the results of employment outcomes for either formally or informally trained AWD. Such information will point to whether there is a particular indication of employment success resulting From either type of training. For example questions such as 'Did formal training lead to better employment rates than informal training, and vice-versa ?', or 'Did the type of training lend itself only to the same type of employment outcome (e.g. formal training = formal employment) ?' l6 Of Ihese, 6 percent or 39 had to be retrained when placed in a business environment A short analysis of these questions was made from the data as presented in the following table. Relationship Behveen Type of Training and Type of Employment Outcomes: Gender Disaggregate iInformal to Formal 20 35 65 1 -- 7i ,Informal to Informal 90 / 35 1 98 1 38 1 I88 1 75 Training to Employment Formal to Formal Formal to Informal As mentioned earlier 36% of all AWD undenvent formal training while the remaining 64% did informal training. As for employment rates the only reliable record of certain employment are for those who have had some form of continuous employment since the end of their training. Overall, 74% or 384 individuals had continuous employment from the time that they finished training. The remaining 26% were either underemployed, unemployed or sick. Of those who undertook formal training 68% had continuous employment while for those who did informal training 76% had continuous employment. This is expected as many who did informal training did it within a business that was reasonably committed to consider employing AWD after the training. That is. the business had a sense that if they were taking on a trainee then they were preparing someone to work within their own business. Not so for the formally trained. These individuals had to go fron~ graduating at an institution to find a business that would employ them in their local community, which is much harder. Therefore, the higher continuous employment rate for informal is perfectly understandable. Having said that. there is not that much difference behveen continuous employment rates behveen the type of training. One would have expected a higher difference. In particular there was a higher continuous employment rate for those that were informally trained. Male 22 50 The proportion of the 384 who were continuously employed who had done formal training was 34% compared to 66% who were informally trained. The figures reflect the sheer numbers of those that did informal training over formal training and proportionally these figires are very similar to overall 4WD trainee percentages (this equally applies to gender disaggregated data). Formal to Formal Of the 131 individuals who have had continuous employment resulting from formal training 40% were employed within the formal sector. The formal sector being their training centre. such as Job Senice Centres, or factories either in their own province or even as far away as the two main cities (Hanoi or Ho Chi % I7 38 Female 30 29 % 23 -- 7 1 Total 52 79 % 40 60 Minh City). On a gender basis 17% of formally trained males had continuous employment in the formal sector. There was a slightly higher percentage of females at 23%. Formal to Informal The remaining 60% who trained formally have gained continuous employment in the informal sector (i.e. small businesses). That is that have gone from the formal training centres into small businesses usually within their own locale. On a gender basis 38% of formally trained males had continuous employment in the informal sector, for females it was 22%. The proportion of formally trained females that went into the formal and the informal sector is equal though for males there are considerably more formally trained that went into the informal sector. There is no empirical explanation for this but anecdotal evidence suggests that females are more 'employable' in 'sweat-shop factories' and that the proportion of females that had done factory orientated training (e.g. sewing) was far higher than that of males. For example, there were 237 AWD who underwent sewing training 78 % of which were female but only 22% male. Informal to Formal Of the 253 individuals who have had continuous employment resulting from informal training 25% were employed within the formal sector. This is a relatively low number as it is expected that if an A\YD trains within a local business that they are more likely to stay within that sector in some capacity. In fact it is a little surprising that the percentage going into formal employment is as high as it is. The explanation may be simple. Factories pay more per month than do many small-businesses and the lure to the major centre for employment provides opportunity and higher wages. For example, nearly 15% of all X\VD from Quang Nam and Danang went to Ho Chi Minh City, most of them to factories. As expected, and as reasoned in the last section above, twice as many females (45%) who were informally trained went into forn~al employnent than males (20%). Informal to Informal 75% of individuals who had continuous employment from informal training remained in the informal sector. There were equal proportions of male and females. This high number is logical considering that many would have gained employment through the same small business in which they had done their training. If income from small business had been more competitive with factories then there would have been less 'leakage_e' of AWD out of their local community. That is, it is probably not for the want of work that the 25% went to work in factories or outside businesses but more due the lack of adequate or attractive income being offered by local small businesses in comparison. Summary 76% of all AWD who were trained informally had continuous employment while for those who undertook formal training it was 68%. The difference is not great. From these figures there can be little certainty that informal training leads more directly to better employment outcomes. A greater percentage of informally trained AWD should have had continuous employment if the logic of employment resulting from the place of training is considered. No definitive conclusion on employment outcomes from type of training can be made from this data. Longitudinal research must be undertaken to gather post-training employment outcomes to get a more accurate indication of whether there. is a type of training preference for employment retention. This is outside the scope of the evaluation and the normal work of project staff. A greater proportion of formally trained AWD gained continuous employment from the informal sector (60%) than from the formal sector (40%). In fact the employment retention rate in the formal sector is mostly due to work gained in factories. If this option was not available then the proportion gaining employment from the informal sector would be much higher. Or more likely. the rate of underemployment or unemployment would be greater as there is a limited pool of informal employment opportunities. A greater proportion of informally trained AWD gained continuous employment form the inforn~al sector (75%) than the formal sector (25%). The surprise here is that there is such as large proportion June 2003 Pagu 35 :\\.TP.-\\\.D E\.h-:iw of informally trained seeking formal work (this is explained in terms of factory work. as above). Clearly, the informal job market is too limited andlor the formal job market to lucrative - Gender disaggregated data adds little to understanding the type of training relationship to employment outcomes. There was around 70% for both sexes in gaining continuous emploplent from formal training. For continuous employment from informal training it was higher for females (86%) compared to males (72%). The only issue from this data is that females trained formally or informally have higher levels of continuous employment in the formal sector. .-\n explanation for this is offered in the sections above. Conclusion Slightly better employment outcomes for informally trained over formally trained but the difference does not seem to be significant. Factories offered less than half of the employment opportunities for formally trained and a quarter of opportunities for informally trained. Small business were by far the biggest employer offering 60% of employment opportunities to formally trained and 75% of opportunities for informally trained. Informal training did not necessarily translate to informal work (only in 75% of cases) = Women do better from formal opportunities perhaps because of the high nunibers required in clothing factories and more females doing sewing training. 6.0 SUSTAINABILITY There is some evidence that local government counterparts are learning from WCDO's interventions in this area: during the evaluation the team leader visited a home where a former project beneficiary was recovering from surgery on her club feet (though surgery is not actually a Project component). The local Red Cross had become aware of her condition and used resources from the American Red Cross to pay for the surgery. Such examples could he interpreted as indicators of sustainability and increased initiative from local players. However, conclusive statements about sustainability are difficult to make at this juncture. It is clear that many people have been exposed to AWD and related issues through this project and self-reporting indicates related changes in behavior. Government officials, especially in the more rural areas seemed enthusiastic about what they had learned and the influence of the project on communities. However. there is yet no visible government-related structure or movement toward assuming responsibility for the project activities. nor any indication that activities would simply be absorbed by the government agencyks responsible for PWD at the cessation of the project. So in this limited sense of the concept of 'sustainability' (ie. continuation andlor replication of activities beyond project Life and external funding), the project is not sustainable. However, if a broader notion of 'sustainable' is applied, in the sense that benetits and increased awareness will continue and will shape the future behavior of people in the target area then the project can be considered sustainable to a limited extent. The DSC in Danang did indicate that through the NGO Vietnam Association for the Handicapped, similar activities were being promoted in Danang; however without the attention to placement for trained AWD and for PWD rather than specifically targeting adolescents. One purpose of the evaluation was to determine whether or not this project could provide a model for other agencies within Vietnam who want to assist PWD in the area of skills training and employment. Thus the evaluation team isolated some key features contributing to the success of this innovative project: Volunteerism and active engagement of the Community Support Workers at -gasroots level in encouraging and facilitating training and employment. Care in placement and monitoring of training component. Longer duration of the training (about 30 percent more time is required than for a 'normal person'). Good selection process for businesses at the start. Most of them seem to have been selected through relational links: known to the AWD andlor family, a friend of the CSW or the WCW social workers. This personal linkage appears to have provided an extra measure of responsibility and accountability. In conclusion, the evaluation results suggest that the model used for raising awareness, vocational training. employment and income generation, and social inteagation and support for AWD is valid for the semi-rural environment of Vietnam. Particularly, the use of the non-formal sector for wcational training and employment was found to be effective and should be regarded as an innovative approach that could help the economic future of PWD in Vietnam and have a positive influence on related policies. RECOMMENDATIONS 1. Management Assign at least one full-time social worker per project site (ie. per proxince) to facilitate capacity building for the CSW's and to ensure quality attention to social issues. Project Coordinators should have full information, including financial information, from the start of a project to manage effectively. This is even more important when project sites are quite distant from one another. Project Coordinators indicated that they would have appreciated more skills training for themselves personally, in general project planning, monitoring, and specitically related to conducting KAP. Project planning should have included more attention to development and maintenance of an adequate information management system, including more deliberate exchange between project sites in order to foster in-house learning and speed the leamingcwe. In future, World Concern should take more care to utilize the information that it collects on beneficiary populationls -or to take a more basic approach to collection of information. For evaluating Phase Two it is important to ensure that WCDO has baseline information against which to assess change over time. Quality of regular reporting could have been much improved - to provide an actual monitoring function rather than just a collection of anecdotes about activities. Regarding GIK, in future concerns expressed by government ofticials about quality and appropriateness of the goods and materials received (which could adversely affect key relationships) could be minimized through having -geater consultation with Project Coordinators about the type of supplies and resources to be provided. and a more deliberate methodology for distribution. 2. Cooperative Relationship = Keeping official minutes of meetings with the various committees (PMT, DSC) would provide a useful reference in future regarding issues and decisions that have been made. and contribute to continuity. It also promotes greater transparency in the communication and decision￾making process. = Capacity building for the partner should be a Project priority - the Red Cross members would have greatly benefited f?om greater attention to building their skills and capacity for community mobilizing and other aspects of the Project. 3. Awareness Raising Currently, AR is primarily aimed at direct beneficiaries (AWD, families, businesses) - and in a sense these are people already sensitized to the issues. Greater effort should be made to widen the circle of influence. In AR, greater effort and more strategic thinking should go into involving successfid Pi?) to advocate for themselves. They demonstrate more clearly than any other 'media' that PWD can be valuable and contributing members of society. For example, the Quang Xani Project Coordinator has recently begun to involve a man in a wheelchair from Hoi .hh. in promoting the Project concept and awareness about disability issues in his community. The entire awareness raising and educational effort would benefit from strategic thinking that would develop a few basic ideas and key messages which can easily be pron~oted and then change in knowledge (and perhaps attitude) would also likely be more easily monitored. 4. Vocational Training Education/literacy should have been more systematically planned and implemented in order to ensure that all AWD had minimum literacy and numeracy skills to ensure success in training. = The Project staff could have done more to assess employment and income opportunities in semi-urban areas (such as agriculture or food processing) such as Quang Ham. 5. Site Improvement = There was considerable underexpenditure against the budget for site improvement. Gixn the fact that the project recognised this need, management should either have been more pro-active to allocate funds for improvements to informal businesses or perhaps considered allocating it to homes of AWD or to other public buildings such as schools which do, or will in the fum. senice AWD. = In some business locations, the evaluation team observed that there could hax-e been improvements made to working conditions in general (\:cry poor toilet facilities. poor li*tins. unhygienic kitchen facilities, etc.) - In those cases, perhaps a cost-sharing arrangement with the owner would have provided a good solution. 6. Employment and Income It would be useful for WCDO to conduct research into the question of whether or not AWDPWD are less productive than their 'normal counterparts' and whether or not it actually requires more time for an AWD to obtain comparable skills with a nondisabled person. This is certainly the dominant stereotype image of them in the minds of informal businesskaining sites. If the perceptions are erroneous, the project would be planned differently and perhaps AWDRWD treated more fairly in future. 7. Peer Group Development and Support = Peer groups appeared to be too dependent on WCDO: more effort earlier in the pmess should have been spent in getting these ~oups tobe more self-sufficient. This is now the focus in Phase Two. Life-skills training should have been more systematic to ensure basic knowledge in key areas. CWS's may also need to be trained in both pedagogy (how to teach) as well as in the specific content area. = Engaging the Peer Groups in the planning and various aspects of monitoring would have been one usefid way to develop ownership and responsibility as well as improve the quality of the project. They know best what barriers they encounter in society. 8. Networking Deeper level of analysis by staff of WCDO project results, and presentation in accessible form. would make an excellent contribution to national-level discussion, perhaps influence the practices of other agencies who see the effectiveness of the informal training and job placement approach, and perhaps even policy. - More deliberate effort to share experiences with other agencies with the express intention of learning and applying learnings would be useful for improving the Project quality (such as the Disability Forum). 9. Project Conception and Design Project scope was not optimal, and not sufficiently wholistic. For example, it did not include prevention measures nor was much effort made to incorporate the community into the Project. Corrective medical interventions were not part of original desig; but cenainly an important feature that can in some cases do more good than vocational training for impnxing a child's opportunities for education and livelihood. It also provided a very visible example of the value placed on people with disabilities. It would be instructive to know the types and proportion of disabilities within the WCDO population that are preventable. Such information would be useful in desi-@ng a prevention component to future disability projectls - and in the immediate term, could be used to educate communities in prevention, treatment, and care of conditions that lead to long-term disability. In the Vietnam context, for a project working to employ people in semi-rural or rural areas. a minimum of three years seems a more rational timeframe for a first cycle of about 100-150 beneficiaries per project site. Little attention or resources were included to improve home physical situation although site improvement was a need identified by the project (so the Project understood the principle of improving access, but did not apply it consistently). There was significant scope for greater involvement of the beneficiaries in planning. goal setting, identifying indicators, and tracking results. Also, WCDO could have done more to utilize the AWD themselves as promoters of awareness in their communities rather than relying almost solely on the DSC and CSW's. Future projects should ensure that components (budget, plans, outcome) are carefully and logically linked. Also it must be remembered that expectations for accountability (such as credit schemes) have a price-tag (money, time for training, systems development, etc.). 10. Advocacy Ail Project staff should be made familiar with relevant legal framework governing disability issues, and should be equipped to educate others and especially those who are in a position to enforce or othenvise implement positive legal changes (ie. to enforce eovemnient policy regarding tax exemption for employing PIVD). DSC members could be encouraged to advocate for the AWD and PIVD in their communities. ANNEX 1 EVALUATION TERMS OF REFERENCE Organization: Project Name: Project Location: Project Manager: World Concern Development Organization Room 103,521 Kim Ma Ba Dinh, Hanoi VIETNAM Phone: 84 4 771 6231 1771 8083 AVTPAWD (Adaptive Vocational Training Pro,gam for .4dolticents with Disabilities) 3 Provinces: Hai Duong, Da Nang, Quang Nam Wanvick Browne (mobile: 091 200 5981) SUMMARY: WCDO is contracting otit a mid-term evalriation of the Adaptive Vocational Traitrirrg Program .for Adolescents with Disabilities (A VTPA WD). The project is implenrerrted in three provinces: Hai Drrorrg irr the north and Da Nang and Qtiang Narn in the central region of Vietnam. i%e AVTPAWD is solely fiinded by USAID and commenced in October 1998. Initially the project uas on& for hvo years (October 1998 - December 2000) witlr $875,344 direct fiords. Ho~wver, a one-year errension (October 2000 - September 2001) with S250.000 ewa jimds was granted by USAID. A.fiirtlrer rlrree jears of the project was granted in October 2001 tlrrrs rendering the period October 1998 to Septerrrber 2001 as ltalfofthe total project component. Tlrerefore, tlris evalua~ion of the first 3 years (October 98 to September 01) is regarded as a mid-term evaltiation. The evaluation is to be undertalierr quite some time aler the end of the first phase to get more of a tnre picture of project impact especiall~ in the area of trairrirrg and employment sustainabilily. Project activities for the first 3-year phase were rirrdertaken and managed at a district level irrvolring 6 districts from Hai Drrong, 4from Da Nang, and 4 in Qrrang Nam. Tire mrmbers ofdirect benlid-Tcnn Evaluation Rcpon equipment bought with USAID has not been undertaken for some time. Appendix A contains a list of all supporting documents useful for the evaluation. All originals will be located in the WCDO Hanoi office. 2 EVALUATION TEAM COMPOSITION Evaluators External Kila Reimer (ICC Cambodia) as team leader. Email: proidev@,icc.or~.kh The Team Leader is overall responsible for ensuring compliance with the Ternis of Reference. cooperative planning for field surveys, coordinating and guiding surveyors, quality control of sweyor's work. compilation of results, and writing the final report. The Team Leader will help assign specific tasks to the local team based on areas of competency, interest, and experience. The Team Leader will also conduct the interview with various levels of project management of WCDO, government partners, and cooperative organizations. Internal The Team Leader will be supported by the local staff of WCDO AVTPAWD. The Team Leader will decide on what level of support is required from the local staff and how and what tasks will be assi~ed according to keeping the surveying exercise relatively objective. Local staff will be responsible for translation and organizing all logistical requirements for the needs of the evaluation. The local staff will coordinate and monitor the local volunteers who will conduct the AWD and associated interviews. The Team Leader will plan the survey strategy in conjunction with the WCDO Country Representative and both will decide what community-based government personnel will be involved in the surrey process. Few of the full-time WCDO local staff, and none of the community workers, have had any experience of evaluations therefore this evaluation will also be seen as a capacity building exercise with much emphasis from the Team Leader and WCDO CR on participation and consultation. A list of those participating in the evaluation and a short description of their role can be found in Appendix B. It is the WCDO CR responsibility to ensure that the evaluation process is kept on track and all requirements of the ex-aluation field and research work are completed according to schedule. Details on fees and expenses are covered in the Contract Service Agreement (CSA) between WCDO and the external evaluator. In general, WCDO will cover all direct expenses (travel, accommodation, materials. etc.). Ail expenses must be accounted for and ori~nal receipts submitted to the WCDO administration. Any incurred expenses will be reimbursed on submission to WCDO of original receipts. A flat fee for the external consultants services will be paid after the submission of the final evaluation report. 3 PURPOSE OF THE EVALUATION The purpose of the 'mid-term' evaluation is to assess the project implementation, participation. results and sustainability of all of the 4 major project activities (Awareness Raising, Vocational Training, Eniployment and Income Generation, and Support Nehvorks). In particular the evaluation will determine whether and to what extent the specific project objectives have been achieved (Appendix C) as well as the detailed project activity outcomes (Appendix D). Specitic results may help determine any structural hindrances that may have determined less than satisfactory outcomes. The evaluation will not only focus on the outcomes but also gauge whether the results sugg,est that the on￾going project is being implemented in a manner that maximizes benefits to the A\W, their fanlilies and community. Also, it will gauge whether the project entertains forms of sustainability. The 'mid-term' evaluation results will be available in time to adjust strategies to help secure employnent and income for the present phase of AWD beneficiaries. Significantly the evaluation will assess whether the model used for raising awareness, vocational training and employment and income generation, and social inte,pation and support for AWD is valid for the senu-rural and rural environments of Vietnam. Particular focus will be upon the model for vocational training and employment especially in the non-formal sector. If the training and employment model for the non-formal sector is a successful one then it may well be regarded as innovative in helping the econonuc future of PWD in Vietnam and could go on to influence policy and decision makers. The summary of the evaluation framework is as follows: PROJECT IMPLEMENTATION Ascertain whether or not the stated 'Project Activities' have been accomplished to what extent and on time. Determine any bamers that have hindered the full accomplishment of any of the project activities. PARTICIPATION AND MONITORING Assess the effectiveness as demonstrated by the organizational structure (i.e. \VCW and go\?. partners) in implementing project activities Assess the monitoring structure and its effectiveness as demonstrated through the implementation of project activities. Determine the value and importance of the government partnerships. at all levels. in implementation and monitoring. Also, identify any hindrances in partnership management that have affected the accomplishment of activities. Determine the effectiveness of Community Support Workers (CS\W as demonstrated by monitoring and support in relevant project activities. Determine the level and value of 'grassroots' community participation and as demonstrated by the accomplishing project activities. Determine the role of the AWD themselves in decision-making and whether the pmject has enabled them to be in a position to ' ... address their own social, vocational, and economic needs'. PROJECT OUTCOMES Ascertain whether the specific Project Objectives of the 4 project components (.-\wareness Raising, Vocational Training, Employment and Income Generation, and Supportive Xetworks) have been fulfilled (see Attachment C) and determine the main factors that have prevented any of the Project Objectives from being fulfilled. Test whether the specific outcomes from each of the project activities have been fulfilled (Attachment D), to what extent, and determine any constraints that have prevented full activity outcomes. Determine any significant difference in results between the formal and informal approach to vocational training. SUSTAINABILITY 0 Determine if there is evidence of the government partners adopting management strategies learnt during the project. Also of any evidence that they have learned and are attempting to implement awareness raising strategies, PWD training and employment models, and support for local PW?> groups and networks. 0 Determine if the general community is better informed from project activities and are, or intend to, replicate awareness raising, vocational training, employment, and social support network activities as modeled by the project. 0 Determine if the AWD and PWD community have been affected by the project in such a way to feel empowered to start their own efforts in awareness raising, vocational training. emploqment. and social support network activities as modeled by the project. 4 SUGGESTED METHODOLOGY The Evaluation Team will focus primarily on collecting basic quantitative and qualitative data to verifu measurable indicators of project activity outcomes. Sampling will sometimes be more purposeful than random because of the diversity of project beneficiaries and activities. The planned methodolo~ is as follows: Review all available project documents including project proposals and reports. Interview WCDO project staff about project management responsibilities, divisions of tasks. and functional \,alidity. Also review their perceptions of effectiveness, structural validity of specitic WCDO management and partnerships. Randomly select a number of PMT (Project Management Team) and DSC (District Steering Committee) members for interview to gauge their perceptions of project structure, management. implementation and results. Randomly select a number of CSW membem for interview to gauge their perceptions of project structure, management, implementation and results. Randomly select training institute leaders, vocational teachers, educators, training and employment businesses, and members of the local community to gauge the benefits of awareness raising activities, vocational training, employment and support nehvorks as well as ascertain the 'participatory' nature of these activities. Conduct random surveys amongst AWD beneficiaries and their families to gauge the benefits of awareness raising activities, vocational training, employment and support nehvorks as well as ascertain the 'participatory' nature of these activities. A\ TPAU D Dlid-Tcm? E\aluat~on Repon 5 SCHEDULE OF MAJOR EVENTS The Schedule will be arranged between the WCDO Country Representative, The Project Coordinators. and the evaluation team leader at round-table discussions nearer the time of the evaluation. The reason for this is the many vagaries that occur while working in Vietnam, as it is hard to establish district cooperation until closer to the time of filed work. EXPECTED OUTPUT A verbal debriefingfeedback session with WCDO staff, after the fieldwork and review preliminary findmgs/results/recommendations. Verbal and written constructive feedback to WCDO about the strengths and weaknesses of the project and any recommendations that may assist better practice in the new phase as well as strategies for locally mandated work in the future. Validation of the vocational training and employment model (particularly informal) that may be repiicable throughout Vietnam and any recommendations that would help the INGO or local community implement such models. The final written report, including a summary in MS Word & Excel and subn~ined to WCDO in soft copy via e-mail. WCDO Vietnam will fonvard final copies of the evaluation report to WCDO Bangkok. WCDO Seattle, and USAID Cambodia. I ANNEX 2 ! OUTPUTS FORM THE FOUR MAJOR PROJECT GOALS Output indicator Output Indicator 1.1: 4t least 7040% ofparticipants in Awareness Raising (AR) haining demonstrate increased zwareness of disability issues. Output Indicator 2.1: 4t least 75% of A WD who sam.cipate will successfully complete their coursds. Output Indicator 3.1: At least 70% of graduates are meeting their vocational needs. Dutput Indicator 3.2: 4t least 70% ofgraduates are zarning incomes greater than that before the training. Dutput Indicator 4.1: 4t least 70% of graduares feel I they receivegreater mutual support than before the program Statement of Achievement More than 8O%of participants in the project who reported attending formal Projict trainingis also reported a change in personal awareness about disability issues. However, not all could provide examples of resulting changes in behavior. Just 41519 AWD did not complete their come (1%): 99% successfully completed their formal or informal course. Only 16% of AWD interviewed indicated they were working before the project. Mostly it was menial labor for meager income (making hats, shoeshine. animals care etc.). According to project records, the employment rate as at the end of March 2003 was 86% employed. During interviews, 68% of AWD indicated they were currently employed (9%in factories and the rest in private business or self-employed). 17%of A\VD interviewed responded that they would choose a different skill if given another choice for training; 22% indicated that they did not enjoy their current occupatios-boring, unsuitable for disability etc The 16% who had work before the project in menial labor to the, reported monthly wages ranged from S2.50 to $130 with most averaging around SlOlmonth. According to Project records, the overall income range for those employed as of March 2003 was S2.5- $4O/month in Quang Nam: S3.5-S55lmonth in Danang: and $3.5-S65lmonth in Hai Duong. The upper level of income range rose S5 for each site, over the time reported (JanIO2-Mar%). 86% said that the program had helped raise awareness about disabilities in their families. Most mentioned more caring, more understanding. sympathy and help, better communication. Only 8% of AWD respondents said they did not regularly attend the Peer Group Meetings. Those who met, mentioned that it was for social company, games. music, sharing experiences. The group activities had helped in confidence. social interaction, fun. happiness, and health learning. Means of verification Intenie\\s during evaluation. Project Records. Inteniews during esaluation. Inteniens during e\aluation. Project Records. Intenisws durin: evaluation. U AVTP?\\\-D Xlid-Tcmi E\nlunr~on Rcpon 67% responded positively when asked if Support Nehvork activities made a difference to them: they mentioned group activities, camping, mutual sharing. Significantly of those that did respond over half Output indicator I mdntioned 'life-skills' teaming asimponant. 1 100% of those emploved (86% of the total) are, of Statement of Achievement Output Indicator 4.2: At least 70% of graduates feel that they apply new knowledge and skills .. . course, applying new skills AWD were asked if there had been a change in their level of social interaction andlor personal health during the Project (poor, fair, good, excellent): 1% said 0 change in social interaction; 33% said I level of change; 37% said 2 levels of change; and 29% indicated 3 levels of change. For chanzes in personal health, 32% indicated 0 change: 39% indicated 1 level change; 24% has 2 levels of change; and 5% had 3 levels of change. Means of verification Inteniens during evaluation. June ZOO3 Past -l? .i\i?\\*.'.D i\,?t;?~ii~m fa I ANNEX 3 1 SUMMARY OF ACTUAL PROJECT RESULTS AGAINST STATED EXPECTATIONS Mid-Term Evaluation: AVTPAWD I Irepmented as desired I Panicipation of \Gous lntenims with P.\IT. DSC. memk of PhlT and DSC not SWs. and PCs. mcture between SC, and CSWs I I Monitoring PhlT and DSC OBJECTIVE AWARENESS RAISING mngest li in the pmjm and provided mosl consiaent Some unexpected delap in ACTIVITIES 1.1 Program ~~~~~~~~~t Coordination Quanerly rrporrs fmm DSC EXPECTED RESULTS Formation of P"Tand DSC and PhlT for \VCG about pmjm xhidcr and mulls wmc fmm CS\W5 thmu& wlx monthly ~poninz. Somaimes the DSC included sdditional information. Infmtion bmv- \\TW and DSC \\ac primarily ml. Rsports no1 tiled wlub b>- the PMT or the DSC - howcvu. indicators Of achievement Participation of allWies nith DSCr. Obra\ed rrp~ls at 2 CSW olliecj; h-ed dcamals in ACTUAL RESULTS PUTIDSC formed in each PmvincriDislrict. Composition not exactly as pmwribed by WCDO in relation to edu and PWD: DocumentationE~idence Intenin\- nith PhlT: intcnim with DX: inrminv nith PC% intmicw nith WCDO CR. AVTP:\\VD hlid-Tvm~ Evaluation Repan trengthenin Cooperative Lelationship Building Trainings and Some PMT and nhm DSC lntcnicas nith PMT. DSC. pacity of PICIT, gender quit). membm inlmieweed indicated that and PCs. ;C some repmentaure of their group had attended at I& WCDO mining - however. not all membas had. Gad= quit).: assuming hat hjm mwdr: this refa to qua1 rep-mulion at hlmgmtPMTS;DSC the PMT and DSC level the comminm are nor muiuble. Of 15 PUT mmbm. 4 arc fmle Of 28 DSC membm in Has kg. 8 arc female Sone of the 12 DSC membm in mg an fdc. Of e 16 DSC membm in Qvang Nam. 2 are f~mle. Thus the o\.nall ratio ofmen to women a the uppm management level (excluding WCDO) is 61/14 or I. about 25% wmm. ust I It75 peopleare PWD: ofthe .~ . . ,.- .. . . ,.. . . . .-. . . ... . . . . . . . . . . . .. . . . . . - . . . . . . . - . . . . -. . . . . '.. . i .. ,-- . .,., .: . Inlbnnxion 1 Sumber uf I DiHicult to aumtih. . . thc nvmk 1 Ouutnh . rms: . intr~\irx acing to impmve meetings with of meetings. as most do not happa nith Ws. diaussion with jpm design other agencies to qularly. The tw places thal cumnt \VCDOCounp d modifications review work Pmjecr Coordinators repaned Rrpmutive; DiubiliIy sharing informadon with the Forum Seuslcctcr. intention of impmving pm-eram design at \sac the Disbilit). Forum and in Iwer ford such as the Seminar on Emplo)ment of Persons with Disbilitier (hang. Au@J). Ser.eral PhlT!DSC meinbas intaviewed indicated uht they wm doubtful that the informal minine model would be swcesful Numk of hang oftice kept ?stemtic Quanaty ~pcfts: inhniol￾rfmls and % words of rrfa-alr. and repancd a *ith FCr: intaviov with DSC. 'ollow up total of 9 during Pharc Onc Ah here is some mrcdoul cridmc in quanaly hportj and specific inslmcer of medical uralmcnt wme mentioned during inlaview uith the PC as well as DSC membas. assislance to WCDO AWD in /pmjm~ for medical mica. 1 ncarparated pmgram lhvl optimal as \\as their .. ". &.+g&6.?g$g&&g<~:$ . . .> .*% ;&.$:I @:m 2: l.*>;&;,:~.~~:~Fg:;~~~;:j;~:*; $I!<;. < >?'.; ..+../.*.+2;-. k.. ;:: 1.3 Baseline Baseline dm for Four prapnm Some bvclinc d313 HX coll~~tedl Intr~li~\ uith PC: dkwi~n no winen record of their conuibution. Situational valuation Analysis 1 was not analrd thomuehk. uu~s No special mrdy \%as conducted on the impact of the Social Workem to the pmjcft. although it is me that CS\V and \\'CDO gaff remembered some of the life-skills lessons taught by the Social W"&rr; pmgram tap&. and \&as not k )stematreally recorded in a %my Intmicw with PC: intminrs with CSWs. /that would enable comparison at I " - artivitirs refined act~v~ties mdoned to program acticitia &a mult of uith CR. mim oipmjm had on data BSATNA Ilat~ collrrtton and nd~vont. Raiew of viws mmnis: intervie\\- with FCs. 1 4 Ipmjrrl end. or impossible for 'paniciwtion. MI this information \va gender dis-amled. Pmm Refined i Them was no discemable chance I I Inteniou nith PC: diiusion A Summary Tablc from the BS&TNA $$as a\ailable fmm analpis analpis. The original project pmpoul remained the major planning % refacncc document lhroughout implementation of Phare One. Ra,pm AWD idmlified Pmgram bencficiaria wme participants exceeds numk identified based on information identified required for collected thm@ the BSA and program ( gender 7h'A swep: diuscgate) The numbm of acNl beneficiaries (519) slightly exceeded the numbmspxified in the pmposal(500). Of the rowl. 262 (51%) wem fmwlcand 257 puang Nam 81 hang indicadnz ,the numbm of AWD sun.e\-ed and then the numbm concluded la be lntmiew uith PC: discGon \\ilh DCS: dijr&on aith CSUX 1 PmjectmMds AVTPA\\'D Mid-Tcne Eraluatioii Rcnon lwme whi joined te p~j& later. / uammo - war mare in the farm of hands-on, or on-the-job mining and exposure to Pmjst id- and i I [forms. I I Repons of I Mort of the CS\Vs intaimved I E\alu;ltim inlmic(\s. pmvide exampla of how the pmject and .AR mining had Teaching gaff proved KAP mmmg. not all look a KAP tat I and - KAP tat rrjulrz were not % greatm KAP Ianal>red or stored to allow for 1 Assuming this refers to teachers lntminv nilh PC% employed for the ABE componml. or for those involved in raational minine. not all meived AR i dcrrtanding I 1 and oRm fmm WCDO swR 1 they mcived visits from the CSWs I [~utw comparison. I 1 Ilikewise. all CSWs intm-imed ! Families with ater % visited by CSW Gmml mmuniry itude change IM)% of families intm-ieu-cd duringthe eralrwlion indiatedthat Inren~iewr wirh families: # reporling ponieiporing in mining: +,reporling allimde chonge Random ample before and aRm program Elalu;ltim intenie- RsulK indicated hat their vkitr to ADW and AD\'/ famititi uasa priority and that they did this regularly. 46% of families intmimvcd rrponed having specific AR mining but fmv could remember dails. 86?6 of AWD interviewed wid the pmjm had helped to raise family awarenss about disabilitic; More ddailr. with erampla of behaviourchange. are contained in the narrative e\;tluation rrpon. It was not possible dluins the evaluation to take a random sample of communily memb lo s\wrs their level of a\mxmas due to gormmmt reruictim and lack of timr Funhamom. there \-+--i+ .-- .3 Adapting Training Institutions odd PWD dav: did the O\.mll. FESYI~~ u.erepmmising - most mentioned lming about disabilities hugh ms\s media (television, radio. n~spaper): mon could give e~ampler of disabililier: 82% of hmilics said that disabled children should and homul school'. Ow half rcmonded with samo ~dcar about how ~XLTS could br. impm\ed for the mobility- ~mwired shoring romc dcwof id study Le skill that they \xmt& . ... . ......... ......... . , Vtable tninina, I I The survey did not actually pe;e the auertion: wuld sou ~refm 1 fornil or infoml &in& to the AWD or their families: the form did includc information about whether or not the AWD would be caplble of, int-led in living awav from home. Aceordine to the ......... ,\t Itat 90 96 oubt about ntwihm the informal JS3RnCSS 1 ... ...... . .... - . :. . . During evalualmn. ???boT.%\VD idcmlirrl xc rumhle to nerd, of AWD &duated the modifications to mining imiluter. Them is also no evidence that PWD were consulted about the tp% of modifications that might k appmpriare. indimrd lh3l thr? \\me not complctzly utirfid nith thr.ir . skdls Ininin8 mplo~mmt Ibonne. ! 1999 Cor ? 3 pjm sm Fin4 graduation for last faml PWD identified as potential wxatioml lechers ouna in June 2001 miJune ~OO?. Accrjs to re- I A number of I WCW staff indicated hat none Quanaly repom conrain some anecdotal evidence that changes %,ere generally appreciated by the few AWD that had need afscees to them in the formal mi"ing cenbps. Then: is no evidence that P\VD were spffifically sou&t out lo be vowtionat teachers within the formal mining cenms. Then wae. aeeord'ig to inteniew with PC and with Centre Dimtor's, at least I PWDas uaina in morl institutions but not ipeeifically mnrited hueh thisoroiect and Icontinuing with I PWD to meiw [clmbmof I WCDO did nM intentionally reparation - officials. ABE teachem trained in dinbilily a~mmrjs. X showing gear KAP. % ustainability extended minine time. on-- of he 5 lechers who tau@l ABE for WC in Phw One received AR training. - one anention lo PWD. or giving individual anention tojob placement mi followup aRa mAiix). 11 should be noted tha~ The innomlive model of mining will be adapted by those mined in the Job Swim Cenms am rcautrrd lo have 3 cmain I pmponion of PWD among tninee; VXAH in thang u-as ~poned by the PC m be engaging in I! agenciedgmup either modeling or expressing interrrt placement or pup suppon Lctivities. hang. and 3 in Quang Nm. There little evidence ihat otha agenciei wen anempting to implement similara).leprojeeLk Howera. DOLISA and Red Cmn Intminv nith PC's A\'TPAi\'D \lid-Ttnn E~alunrion Report aria Income ave nnpmved contml pup lemployed all 3 h&es . I Generation I' neome I showed a s~teht -. tmomremmt cn the/ uppsr range value in the fan quana (Jan-hlmO3). I I WCDO did n~ nstematicalk / the swtus ofa conml goup wing Phase One. I Cmployment and Income I AWD are in I AWD rewns on/ Pmiect records indicate r hi& I Euhmion intmims ~ith thth mng postion lo cumnl vocational Irtrl of rmplo)mml(86?b) a 21 .\\VD's pwjm RY&. ~mpm%elheir land economic l\;kh 2003. Asummg that the I 2.5 AWD employed existing Emppyment usinejses and konomie life iposition 1majorit)- did no1 hare an inwmc I Test of income \Vhm reviewed ova a 5 quans levels compared lo period (January 2002 -March baselinedata ad a 003). the ineomeransesof prior to mteing rhis project. it is funher aswned that they ax receivine imomved income if fin. data to allow for cornparision. Project records indicate that 74Y0 of A\VD in the pmgam u.m continuously employed aflcr completing their wining course. with 2% ddenitely unemployd and thc mplo)mmt satus of agrncy or ministq. orolhm DEI.ISA or by othzr SGO's (such NCO, indicate that as VAH), but no1 lhe Projwt in ili hey will utiliw tirep. his model. 1 In addition, it uas clear fmm I intervirus that by self-admisriw I PhlT ad DSC membRs were ! impwed with Fmjat mulu skills ad emplo)msnt hu& infoml wetar ratherthan ditional formal mining I inwituliwls) and rtared it U-AS workers and Group Ho\vcva. most CS\\Ps indicated that the?. had rrceived some wm of foml minine. be June ZOO3 Pqc 54 A\.~P~xKD E*..$rxiw Formation CSW raise Account of awarmejs in lwal acti%itia and random communin-durine the - it BSAXXA or AR. There war no random fecdbacksolicitd fmm the Intmiea \Gth DSC: intmior- with CSWs. >\\!TPAWD Mid-Tcnn Eraluation Rcpon ustainabilit) QumerIy repom indicate ctivities which mw in ommunilies. It uar elearht cnvalised level - and occw liviw - and all who did repom appreciated and dued the mmings (for diffmt mw). Reasons wm usually rchted to making ("ends. having others nilt whom to sharecommon I la hi$light. I Grouos I Gmuos continuel hlost AWD intm-ioved aid this1 I 11ife. I Excursions. xh ascamping trips, wze most oRen mentioned as htenie\\-\\iIh AtVWs couraged and ow potrntial to beyond cum1 lgdm in self￾Ip efforts. meeting even aRa the project is ended. pmup was still meefing am aRer Phase One otliticially closed - however. WCDO indicated that CS\Vs and \VCDO mffm still pan of the facilitating of the pea pups so it is not clear how much initiadvc is being &en by the AWD themselves. [X.B. PC's =id that in Phasc Two. much more attention is being paid to muking the wer mum wlf-sutlicicnl I EVALUATION FRAMEWORK ANNEX 5 I I PROJECT ACTIVITIES I 1.2 Strongthonlng Cooporatlvo arrals and % follow Junc 211113 l?#c 57 1 1.3 Baseline Situational .~. Program activities refine~ based on data analysis Awareness - - -- Four program objectives have target figures .. ............... .............. Refined actkities endoked AWD identified exceeds number required for program ( gender d saggregate) Who undertook the 8SA , when and how? ... Who weretne.!a!ge!.9!p,P.?. Neqers 7 Target fig~res from BSA for: Disability ~wareness. ~oc~rayn ng. Employment. S~pport Net. Examples of BSA survey, results, tables 7 .. - - Dio BSA data help refine the .. .- pro3ram ? How ? Raw data on beneficiar es ioent fie0 and gender d fferentiat on 7- . ~~ How many formal vs informal and why 7 TNA was participatory Number of AWD participating in TNA as % of total participants What was the TNA, where are the examples and who constructed it 7 pj ~- I Who conducted the TNA, when and how 7 Results recorded 7 Training institutes > PWD PWD to evaluate modifications Did any training institute receive modifications relevant to PWD needs 7 Where and what 7 Did any businesses receive modifications relevant to PWD needs 7 Where and what 7 Number of AWD who dld ABE -where 7 Number of ABE teachers who had AR training and havo good PWD to racoivo scholc~rshlps fo Numbar of PWD approved for TOT toncher training Woro any PWD npprovod of TOT 7 Was It undortnkon -who, whero, when 7 P 2.5 Employment and lncome eneration AWD employed In existing Test of lncome levels compared to businesses and have Improved baselme data and a control group How many AWD graduates have on I (budget) Did any AWD receive equipment AWD reports on current vocational and economic position How has AWD employment improved own I famii IGroup Formatlon CSW properly trained CSW rake awareness in local AWD addressing needs lhrough group acllvity CSW complete one month training course Accounl of activities and random community feedback Number of AWD atlending moolings, and posllivo foodback roporls How does CSW RA in local communlly 7 How oflon 7 I Nurnbor of AWD peer -- groups formod and running 7 Where, slze. how oflon 7 Whal are - How, whero an Did you follow n llfa.skllis curicula 7 if so what one? (ovldcnco) Groups show lntiliallvo lo go beyond supporl of prolcct and ra