Two-Part Evaluation of the Organizational Viability and the Targeted Capacity-Building Components of The Center for Victims of Torture’s Capacity Building of Foreign Treatment Centers Program Two-Part Evaluation of the Organizational Viability and the Targeted Capacity-Building Components of The Center for Victims of Torture’s Capacity Building of Foreign Treatment Centers Program Jennifer Whatley Senior Technical Specialist, Civil Society and Governance World Learning and Andrew Rasmussen, PhD New York University School of Medicine Bellevue/NYU Program for Survivors of Torture 1 This publication was produced for review by the United States Agency for International Development’s Victims of Torture Fund, DCHA/DG, under contract #DFD-M-00-04-238-00, with Manila Consulting Group, Inc. The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS ACRONYMS AND ABBREVIATIONS ...............................................................................IV OVERVIEW OF THE EVALUATION ..................................................................................V PART I: THE ORGANIZATIONAL VIABILITY OF THE CAPACITY BUILDING OF FOREIGN TREATMENT CENTERS PROGRAM ............................................................ VI 1. Executive Summary .................................................................................................................vi 2. Methodology and Limitations of the Evaluation......................................................................1 Notes on Limitations of the Evaluation ............................................................................................2 3. Overview of the International Capacity Building (ICB) Project.............................................3 4. Findings and Conclusions.........................................................................................................4 5. Recommendations................................................................................................................... 20 A Final Note.....................................................................................................................................22 PART II: CVT’S TARGETED CAPACITY-BUILDING PROGRAMS .............................23 6. Introduction ............................................................................................................................ 23 7. Trauma Healing Initiative (THI)............................................................................................ 24 8. International Research and Program Evaluation Collaboration (IRPEC)........................... 33 9. Relevant Issues Common to Targeted Capacity-Building Programs.................................... 48 ANNEX 1: SCOPE OF WORK..............................................................................................52 ANNEX 2: MEETINGS AND INTERVIEWS.......................................................................56 ANNEX 3: KEY DOCUMENTS REVIEWED......................................................................57 ANNEX 4: TORTURE TREATMENT CENTERS SUPPORTED BY ICB ........................59 iv ACRONYMS AND ABBREVIATIONS ACTV African Center for the Treatment and Rehabilitation of Torture Victims AFESIP Acting for Women in Distressing Situations CAPS Capacity-Building Program CAS Center for Advanced Study CIVIC Campaign for Innocent Victims in Conflict CBT cognitive-behavioral therapy CVT The Center for Victims of Torture DC CAM Documentation Center of Cambodia IAI World Learning Institutional Analysis Instrument ICB International Capacity Building IMLU The Independent Medico legal Unit IRCT International Rehabilitation Council for Torture Victims IT Information Technology IRPEC International Research and Program Evaluation Collaboration KID Khmer Institute for Democracy KRT Khmer Rouge Tribunal M&E Monitoring and Evaluation MSI Management Systems International NYEMO NGO in Cambodia Sach! NGO in Pakistan SPANS Special Programs to Address the Needs of Survivors PSC Psychsocial Counselors THI Trauma Healing Initiative TPO Transcultural Psychosocial Organisation USAID U.S. Agency for International Development USIP United States Institute of Peace v OVERVIEW OF THE EVALUATION This report presents findings, conclusions, and recommendations from a final evaluation of the Capacity Building for Foreign Treatment Centers program, a ten-year project implemented by The Center for Victims of Torture (CVT) through a cooperative agreement with the U.S. Agency for International Development’s Victims of Torture Fund. The overall final evaluation was conducted by a two- person team: Jennifer Whatley from World Learning, and Andrew Rasmussen from the New York University School of Medicine. The two separate reports written by Ms. Whatley and Dr. Rasmussen are combined in the following two￾part report. Ms. Whatley’s primary role was to evaluate the project’s efforts to strengthen the organizational viability of torture treatment centers in multiple countries. Ms. Whatley’s original report is included as Part I of this report, “The Organizational Viability of the Capacity Building of Foreign Treatment Centers Program.” Dr. Rasmussen evaluated clinical and research aspects of the project. Dr. Rasmussen’s original report is included as Part II of this report, “CVT’s Targeted Capacity-Building Programs.” Copies of both Scopes of Work for the evaluation are included in Annex 1. vi PART I: THE ORGANIZATIONAL VIABILITY OF THE CAPACITY BUILDING OF FOREIGN TREATMENT CENTERS PROGRAM 1. Executive Summary The evaluation took place in July and August 2009, and included site visits to CVT, as well as to project-supported torture treatment centers in Kenya and Uganda. The evaluation sought to answer several key questions: 1. What were the project results? 2. What could have been improved? 3. What insights emerge related to the following: 4. Appropriate size of potential partner organizations 5. Ability of an organization to graduate from the program 6. Program support to organizations primarily focused on human rights versus those primarily focused on health care 7. Maintaining a global focus versus a single-region focus 8. Maintaining a holistic approach that addresses organizational development, clinical, advocacy, technology versus taking a focused capacity building approach The evaluation found that the centers appreciated CVT’s support and credited it with improving their overall capacity. CVT’s initial support for the centers in Kenya and Uganda came at critical times when the organizations lacked funding. The Uganda center was “like a person on his death bed” and may well have closed were it not for CVT’s willingness to risk supporting an organization facing serious organizational challenges. The centers also praised CVT’s approach to partnership. However, the evaluation found that it was not possible to determine the extent to which the project’s efforts affected the organizational development of the centers given the information available. The evaluation makes several recommendations for USAID and for organizations implementing similar USAID-funded projects: 1. Improve project reporting by requiring comprehensive annual reports to USAID 2. Develop a comprehensive Monitoring and Evaluation (M&E) plan with clear outcome indicators and targets 3. Revise and consistently apply the Organizational Self-Assessment Matrix 4. Clarify the process through which organizations graduate from the program, and provide benchmarks for such graduation vii 5. Identify regionally based resources for capacity building 6. Maintain a holistic approach 1 2. Methodology and Limitations of the Evaluation This report presents findings, conclusions, and recommendations from a final evaluation of the Capacity Building for Foreign Treatment Centers project, a ten-year project implemented by The Center for Victims of Torture (CVT) through a cooperative agreement with the U.S. Agency for International Development’s Victims of Torture Fund. The program is referred to internally at CVT, and for the remainder of this report, as the International Capacity Building (ICB) Project. The purpose of the evaluation was to assess the outcomes and impact of the project in its efforts to strengthen the organizational viability of the treatment centers, and to offer recommendations that could assist USAID as it considers supporting similar projects. The evaluation sought to answer several key questions: 1. What were the project results? 2. What could have been improved? 3. What insights emerge related to the following: 4. Appropriate size of a potential partner organization 5. Graduation of an organization from the program 6. Supporting organizations primarily focused on human rights versus those primarily focused on health care 7. Maintaining a global focus versus a single-region focus 8. Maintaining a holistic focus that organizational development, clinical, advocacy, technology) versus a focused capacity building approach? The evaluation took place during July and August 2009 and included a review of documents, site visits, and individual and group interviews. Site visits were made to the offices of The Center for Victims of Torture (CVT) in Minneapolis, Minnesota, from July 16 to 20 and to two of the centers supported by CVT through the ICB Project: the Independent Medico-Legal Unit (IMLU) in Nairobi, Kenya, visited from August 3 to 4, and the African Center for the Treatment and Rehabilitation of Torture Victims (ACTV) in Kampala, Uganda, visited from August 13 to 14. During those visits, twenty current or former CVT, IMLU, and ACTV staff were interviewed. Prior to the CVT site visit, the evaluator (Whatley) also met with Cathy Savino, project manager of the SPANS (Special Programs to Address the Needs of Survivors) Technical Support Contract, and Lloyd Feinberg, manager of five Congressionally-mandated funds that comprise SPANS. See Annex 2 for a list of interviews. Prior to and during those visits, evaluator Whatley reviewed numerous documents provided by CVT, IMLU, and ACTV. Those included select quarterly narrative reports submitted by CVT to USAID, select quarterly narrative and financial reports submitted by IMLU and ACTV to CVT, results from organizational assessments of the centers, select trip reports submitted by CVT staff and consultants following site visits to IMLU and ACTV, emailed responses to a 2007 survey 2 which CVT conducted with the centers, IMLU and ACTV annual reports, and other documents. A list of most relevant documents reviewed is included in Annex 3. Notes on Limitations of the Evaluation Organizational viability vs. clinical services – As noted above, an evaluation of clinical services is provided later in this report. However, because organizational viability directly relates to the ability of IMLU and ACTV to deliver clinical services, interviews with staff at those organizations often touched upon ICB’s support for both organizational viability and for clinical services. Therefore, this section of the report occasionally repeats statements made by IMLU and ACTV staff regarding training received through the ICB project related to clinical areas. These statements should not be read as a comment (positive or negative) by the evaluator on the effectiveness of the training, nor on the effectiveness with which skills reportedly gained are now employed. Available written material – One challenge in evaluating a project in operation for nearly nine years is reviewing the sheer volume of information available. CVT provides quarterly programmatic reports to USAID and requires quarterly reports from centers receiving grants. Neither CVT nor the centers produce annual or regular cumulative reports on project activities and results. It was not feasible during the time allotted for the evaluation to review the, presumably, thirty-five ICB quarterly reports submitted to date in order to obtain a complete picture of the project. The evaluator instead reviewed original project proposals, quarterly reports for the past year, a list of grants provided over the life of the project, results from organizational assessments and CVT surveys with centers, and other documents most likely to yield information on project impact. Diverse nature of centers – During the life of the project, CVT has supported twenty centers in nineteen countries on four continents. Those centers vary markedly in terms of size, level of organizational development, resources, historical experience, operating environment, and mission. It is therefore difficult to draw broad conclusions about the ICB project’s support for centers. The evaluator was able to conduct site visits to two of the centers supported by the ICB project. Those were selected in consultation with CVT and approved by USAID. Visiting ACTV and IMLU allowed the evaluator to analyze the experience of two centers operating in roughly similar (though certainly not identical) operating environments, to compare a current ICB grantee with a “graduated” center, to compare a center focused exclusively on serving victims of torture with one that also serves victims of other forms of trauma, and to examine centers operating in a region of the world that may remain a priority for similar projects in the future (given the number of torture victims and lack of resources to serve them). Where possible, the evaluator has supplemented information obtained from those site visits with information related to other centers obtained by reviewing documents and interviewing CVT staff. 3 3. Overview of the International Capacity Building (ICB) Project The Capacity Building for Foreign Treatment Centers program (2000-2010) is implemented by The Center for Victims of Torture (CVT) through a cooperative agreement with USAID. In 2000, ICB was awarded as a two-year project (October 1, 2000 to September 30, 2002), with the goal of strengthening the capacity of foreign treatment centers in terms of organizational viability, the delivery of services, and advocacy of human rights. USAID amended the agreement four times to extend the program end date to September 30, 2003 (no-cost extension); June 30, 2004 (cost extension); June 30, 2006 (program expansion); and June 30, 2010 (cost extension). Total USAID funding awarded over the life of the project now stands at $8,564,684. The project’s initial goal was “that centers staff have adequate skills, tools and systems in place to effectively manage their centers in terms of strategic/program planning and evaluation, board development, human resources, clinical and overall organizational development.” With the 2004 extension, the program expanded to include more focused capacity building with a subset of centers, including the Trauma Healing Initiative with the Transcultural Psychosocial Organisation (TPO) in Cambodia and the International Research and Program Evaluation Collaboration (IRPEC) with six current or graduated ICB centers. The goal of the project as it now stands is to expand available global resources, professional and service capacities, in order to heal and support survivors of torture. The project works to encourage new approaches for the prevention of torture and restore civil society through the strategic position of healing centers. In its current form, the project has four objectives and two sub-objectives: 1. “ICB Center Strengthening”: Strengthen the capacity of global torture treatment centers, in terms of organizational viability, the delivery of services, and advocacy for human rights 2. Sub-objective A: Expand the network of project partners by engaging four additional centers in all aspects of technical assistance and project support 3. Sub-objective B: Continue supporting and providing assistance to the present sixteen project partners, with a “graduation” strategy to disengage with up to ten project partner centers 4. “Trauma Healing Initiative (THI)-Cambodia”: Through a local torture treatment center, TPO Cambodia, build the clinical and public education capacity among a network of community torture/trauma treatment resources in Cambodia to more broadly impact torture survivors 5. “International Research and Program Evaluation Collaboration (IRPEC)”: Continue to strengthen research capabilities of project partner centers to enable them to measure the impact of rehabilitation interventions on the functioning and well-being of torture survivors 4 6. “CVT - Sierra Leone”: Continue to provide extensive clinical training and supervision to a group of experienced Psychosocial Counselors (PSCs) who, in turn, offer direct counseling services to survivors of torture in Kono District of Sierra Leone, while fostering the development of “CAPS,” the Community Association for Psychosocial Services, an independent torture rehabilitation/resource center in the Kono District. 1 Under objective 1, over the life of the project, CVT has supported twenty torture treatment centers in nineteen countries (see Annex 4 for a list of centers). Activities include sub-grants, on￾site technical assistance, monitoring and coaching visits, regional exchanges, training workshops, a newsletter, and networking events. 4. Findings and Conclusions 4.1 CVT’s Approach and Project Values ICB’s goal is long-term, but funding has come primarily via a short-term cooperative agreement and extensions. The project was initially funded for two years, and it was only with the most recent cost-extension (2006) that the project received funding for more than a two-year increment. This no doubt complicated project planning and affected CVT’s ability to develop capacity building plans with centers for more than twelve to eighteen months in advance. CVT’s capacity building approach is flexible and focused on responding to needs identified by the centers themselves. Most capacity building is delivered through subgrants, rather than through CVT-organized training or technical assistance. That method reduces the management burden for CVT, and allows centers more control over the assistance they receive. Generally, the process of supporting a center began with CVT staff or consultants leading a self-assessment process whereby the center’s staff identified areas in which they desired to improve organizational capacity. The center then developed a Technical Assistance Plan to address those needs (see section 4.3 for additional information about this process). CVT then asked centers to submit proposals based on the Technical Assistance Plans. Centers are able to use grant funds for a wide variety of purposes, including purchasing computers and equipment, supporting rent and core staff salaries, funding direct activity costs, and contracting organizations to provide training or technical assistance to center staff on a range of issues. That approach allows partners to use funds where they believe the need is greatest. IMLU and ACTV staff emphasized that CVT funded line items that other donors would not, such as equipment and core operating costs. For example, a 2007 grant from CVT allowed IMLU to purchase a photocopier, replacing a machine that had broken two to three years prior. Both IMLU and ACTV benefitted from CVT’s flexible funding approach and willingness to fund smaller organizations. ACTV had lost its one donor when CVT first began funding the organization. IMLU also had difficulty finding donors when it obtained its first CVT grant. CVT’s flexible approach does complicate efforts to evaluate the overall impact of ICB’s capacity building, because the type of capacity building provided differs from center to center. 1 CVT quarterly reports note that objective 4 was completed in 2008. The other three objectives are ongoing. 5 From interviews with CVT, IMLU, and ACTV staff, it is clear that those organizations primarily view theirs as a partnership relationship rather than a funder-grantee relationship. ACTV staff described CVT as firm on standards, but interested in solving problems and working with centers to achieve standards. Interviews with CVT staff indicate they view CVT and centers as part of a global effort to combat torture. At times, CVT support for centers extends beyond the ICB project. For example, CVT has assisted centers in reaching out to other donors by offering feedback on draft proposals, serving as a reference to other donors, and bringing centers to the attention of interested donors. CVT has included center staff in workshops organized under its New Tactics Project and will provide clinical support to the centers in Kenya, Uganda, and Cameroon under a new three-year grant funded by the European Union’s European Instrument for Democracy and Human Rights. 4.2 ICB Monitoring and Evaluation Plan and Efforts The ICB project does not have indicators for measuring outcomes under the ICB Center Strengthening Objective. The project’s Detailed Implementation Plan for 2006–2008 lists major planned activities and includes output level indicators (e.g., “minimum of three staff exchanges conducted between selected ICB partners”). While useful for monitoring project implementation and ensuring that activities take place as scheduled, those indicators do not provide insight into the project’s impact. The one indicator that could provide some information on impact— “increased ability of ICB project partners to collect data” —was to be measured by surveys sent to centers. Quarterly reports submitted by CVT to USAID covering the period July 2008 to June 2009 did not indicate whether the survey had been conducted or how increased ability was defined or measured. CVT quarterly reports to USAID do not include cumulative reporting on indicators. A review of ICB’s 2004 and 2006 cost extension requests to USAID did not reveal clear plans to measure outcomes. CVT’s 2000 proposal to USAID included suggested indicators that did go beyond counting activities. For example, “Centers will secure new support from institutional and individual donors and reduce their reliance (as a percentage of revenue) on international donors.” It is not clear whether CVT tracked those indicators once the project was awarded. CVT has made efforts to capture information about project impact. Those include using the project’s Organizational Self-Assessment Matrix to capture information about each center’s stage of development (see section 4.3 for discussion on this point), surveying centers, and conducting internal retrospectives. Those efforts provide anecdotal information but not a clear picture of project impact. In addition, because CVT does not prepare cumulative reports on indicators, it is difficult to determine whether activities have taken place as planned without reading through multiple quarterly reports. 6 4.3 Organizational Self-Assessment Matrix and Process for Assessing Center Capacity CVT’s “Organizational Self-Assessment Matrix” has the potential to serve as a tool for assessing the capacity of the center, informing project assistance, and capturing change to the center’s capacity over time. It, therefore, provides information related to project impact. The matrix is modeled, at least in part, on instruments developed by organizations experienced in organizational capacity building. The methodology for completing the matrix has varied over the course of the project and from center to center. While available information indicates that the organizational capacity of centers has improved during the life of the project, the degree to which the ICB project contributed to that positive change is unclear. 4.3.1 Overview of the Matrix and Process of Implementation According to CVT’s Project Partner Manual, the matrix is designed to serve the following purposes: • Identify areas of most concern in the perceptions of the organization’s leadership and staff • Establish general baseline information • Assess the stage of institutional development and the priorities for institutional strengthening • Serve as a measure of monitoring and evaluating progress in institutional development • Serve as a staff development tool • Help to build a shared commitment to organizational development and change • Assess training needs of staff, boards, and committees • Assess the Information Technology Capacity of individual organizations The matrix appears to have been used most effectively to build shared commitment between CVT and the center to organizational development and to determine priority areas for institutional development. CVT has also attempted to use the matrix to evaluate progress, although, as discussed below, this was not an early priority of the project. CVT currently uses the matrix to analyze each center’s capacity along seven functional areas and twenty-eight sub-functional areas as listed in Table 1. Organizations are asked to categorize themselves within four distinct stages of development, using a description of a typical organization at each of those stages for each of the Sub-Functional Areas. By assigning a numerical value to each stage, CVT can then calculate the average of those values to determine the overall stage of development for the Functional Area. Those stages of development are (from weakest to strongest) Emerging, Developing, Consolidating, and Sustaining. As CVT notes in its introduction to the matrix, an organization may be at different stages of development for different Functional Areas—for example, “Emerging” for external relations, but “Developing” for financial resources. See Table 2 for an excerpt from the matrix. A CVT staff member or consultant, along with center staff, conducted the initial assessment with fifteen of twenty centers. The assessment took place during a week-long visit aimed at building trust with centers, determining needs, developing a technical assistance plan, and laying the groundwork for grant applications to come. During those trips, CVT staff also routinely met with 7 institutions that could serve as funders or providers of technical assistance to the center. After completing the initial assessment, CVT helped the centers to prepare a Technical Assistance Plan informed by the assessment results and later to prepare grant proposals to address aspects of that plan. 8 CVT initially used the matrix as a framework for assessing needs and developing plans for assistance, rather than as tool for evaluating impact. The 2002 external management review of the project recommended that “The project should identify baselines within its extensive organizational matrix so that progress in organizational development can be more clearly delineated.” In response, CVT retroactively assigned stage-of￾development scores based on notes from the initial visits to the twelve centers the ICB project was then supporting. When three new centers (Bangladesh, Kenya, and Rwanda) were added in 2002, CVT staff used the matrix to determine the baseline stage of development from the outset. A list of assessment results for each center reviewed during that evaluation indicated that the three centers (TPO Cambodia, Kosovo, and Namibia) added to the project after the 2004 extension filled in the initial matrix themselves and so presumably were not assisted by CVT staff in doing so. The information provided by CVT listed no assessment for the centers in Cameroon or in Sierra Leone. After the initial assessment, CVT periodically asked centers to fill in and return the matrix themselves, generally in connection with applying for a new grant. As noted in the 2002 External Management Review, CVT did not have prior experience implementing international NGO Table 1: Organizational Assessment Matrix Functional Areas with Corresponding Sub-Functional Areas Functional Area Sub-Functional Areas Governance Board Mission Constituency Leadership Operations and Management Systems Planning Administration Program Human Resources Staff Roles Work Organization Performance Management Salary Administration Conflict Resolution Communications Financial Resources Accounting Budgeting Financial Control Audit Fund Management Resource Base Legal Status Service Delivery Sectoral Expertise Impact Assessment Information Technology Information Technology External Relations Public Relations Constituency Relations Government Collaboration Organization Collaboration Local Resources 9 strengthening projects when it was initially awarded the cooperative agreement. CVT could rely on its core strengths to deliver components of the project devoted to building counseling skills, but the organization had to develop new approaches for the project components devoted to improving the organizational viability of centers. To do so, CVT looked to organizations more experienced with providing such assistance. According to the introduction CVT provides to its matrix: A significant portion of the Organizational Assessment Matrix tool is the result of review and, in some cases, significant ‘borrowing’ from assessment instruments developed by other organizations. The “Institutional Assessment Instrument” developed by World Learning has been particularly useful.2 2 Center for Victims of Torture, International Capacity Building Project: Project Partner Manual July 2008-June 2010. This document describes the purpose of the tool, defines key terms, and describes a participatory process by which to complete the matrix. Table 2: Organizational Self-Assessment Matrix Excerpt, Financial Resources Functional Area Sub￾Functional Area Stage of Development Emerging Developing Consolidating Sustaining RESOURCE BASE Funds solicited for one short-term project at a time and only come from one source. Little understanding exists of the need for the Organization to eventually become self-supporting financially and local fund-raising for any income is untried or unsuccessful. Program funding is insufficient to meet plans or provide projected services. Organization can prepare multi-year program budget but still dependent on single or limited donors. Developing awareness of local resource generation possibilities but few funds actually mobilized. Funding is available to cover short-term project costs. No single source of funding provides more than 60 percent of funding. Significant percentage of core costs covered by locally generated revenue (membership dues, fees, regular fund￾raising, etc.). Funding is available for short-term costs and medium-term funding strategies exist within funding plan. No single source provides more than 40 percent of funding. Long-term funding plan exists that will result in Organization becoming sufficiently self￾supporting so as to be able to continue basic operations if external funding is cut back. All projects have long￾term funding plans and current funds are adequate to meet needs of management plan. Basic program delivery can continue even if there is a shortfall in funding that is not self-generated. 10 With the exception of Information Technology, the Functional Areas on CVT’s matrix correspond to Competency Areas listed in the World Learning Institutional Analysis Instrument (IAI), 3 as do many but not all of the Sub-Functional Areas. These functional areas also roughly correspond to those found in assessment instruments developed by PACT and Management Systems International (MSI), among other organizations, and successfully employed in USAID￾funded projects. 4 Like the CVT tool, the World Learning, PACT, and MSI tools also categorize organizations along four stages of development. World Learning has found the IAI process to be useful for a number of purposes, including serving as the basis for action planning, comparing levels of development across organizations, and evaluating changes in capacity within an organization. The World Learning, PACT, and MSI tools are intended to be conducted through a participatory process, which involves the organization’s leadership and staff and possibly outside stakeholders in assessing the organization’s capacity, similar to what the document review indicated that CVT envisioned for its matrix. The participatory process has the advantage of building buy-in within an organization regarding the results and action planning that may follow. With its own assessment tool, World Learning has found that the role of the facilitator is very important in obtaining an accurate picture of the organization. The facilitator’s role includes ensuring that key voices from within the organization are represented, helping participants understand the terms and concepts discussed in the tool, guiding discussions so that participants reach consensus, and challenging participants to explain and justify their determination of their organization’s stage of development. Other than the initial application of the matrix (for fifteen of eighteen centers), CVT did not provide a facilitator to centers when they completed the matrix. As will be discussed in section 4.2.2, this likely affected the quality of information obtained in later applications of the matrix. There are also two areas covered by the matrix that could be strengthened to better support the ICB project: Information Technology (IT) and Advocacy. The IT Functional Area includes no Sub-Functional areas and combines into one category use of technology, number of computers, internal IT configurations, and IT staffing. Given that improving IT capacity has been a priority for the ICB project, as evidenced by its having devoted a specific category of subgrants to technology enhancement and by providing assistance to centers to develop client tracking databases, it would be useful for the project to break this functional area into two to three Sub￾Functional Areas in order to better assess needs and track progress. Also, because numbers of 3 World Learning’s Institutional Analysis Instrument (name later changed to Participatory Institutional Analysis Instrument - PIAI) was developed in the 1990s and by 2000 had been tested in projects in Sub-Saharan Africa, Eastern Europe/former Soviet Union, and the Caribbean. For the purposes of this study I compared the CVT Matrix with the World Learning PIAI as updated in 2000. One difference between the two is that World Learning’s PIAI breaks its six Competency Areas into 28 Categories (for the most part corresponding to CVT Sub-Functions) as well as 91 Sub-Categories. PIAI thus produces a more nuanced picture of the organization than does the CVT Matrix, but is presumably more time consuming and complicated to administer. 4 These are PACT’s Organizational Capacity Assessment Tool and MSI’s Institutional Development Framework. This section draws upon an internal report prepared for World Learning -- Raluca Nemtanu, Participatory Institutional Analysis Instrument: A Comparative Analysis of Similar Instruments, Lessons Learnt from the Field, Recommendations for Future Use, World Learning, September 2006. 11 computers are included in the criteria that determine whether an organization is Developing, Consolidating, or Sustaining, the IT matrix appears to have a bias against smaller organizations. In discussion with CVT, IMLU, and ACTV staff, three types of advocacy were noted: 1) policy advocacy such as ACTV’s efforts to pass a Law on Torture within Uganda; 2) public advocacy, such as working at the grassroots level to raise awareness of the need to reduce stigma against torture victims or awareness raising among security forces to reduce the incidence of torture; and 3) legal advocacy, such as providing legal advice to torture victims to support their claims before courts, government agencies, or international bodies. Centers do not necessarily aspire to conduct all three types of advocacy. Advocacy skills are implied under the External Relations Functional Area of the matrix and its Sub-Functional Areas of Public Relations, Constituency Relations, Government Collaboration, Organization Collaboration, and Local Resources. However, policy advocacy is only specifically mentioned in describing a Sustaining organization and legal advocacy is not explicitly mentioned. 4.3.2 Organizational Change in Centers as Measured by the CVT Matrix CVT has at least two completed matrices for sixteen of the centers. Comparing the average stage of development among centers at the time of their first assessment with the average stage of development among centers at the time of their most recent assessment shows that, in general, centers increased their organizational capacity during the course of the project. The chart on the next page compares the average capacity of the centers in the seven Functional Areas at the time of the initial assessment with their capacities at the time of the most recent available assessment. In order to average ratings and compare change over time, the development ratings for each category are represented numerically in the graph using the same one to seven scale that CVT employed in its 2004 ICB Project Internal Retrospective: 7-Sustaining 6-Consolidating/Sustaining 5-Consolidating 4-Developing/Consolidating 3-Developing 2-Emerging/Developing 1-Emerging As the chart shows, the average stage of development increased by more than one-half stage of development in all seven categories. The largest increase came in Information Technology, where centers improved from an average capacity in the Emerging/Developing range to an average capacity in the Developing/Consolidating range. The smallest increase came in Human Resources. The level of change varied among individual centers. All sixteen centers experienced positive change in at least two Sub-Functional Areas. This change ranged from positive change in two Sub-Functional Areas for TPO Cambodia and three Sub-Functional Areas for the center in Peru, to positive change in twenty-seven Sub-Functional Areas for the center in India and twenty-six 12 Sub-Functional Areas for the centers in Bulgaria and Ethiopia. Six centers reported declines in capacity in at least one Sub-Functional Area. Those were the centers in Guatemala (two), Namibia (three), Pakistan (one), Palestine (two), Peru (sixteen), and Rwanda (two). Shortly after the baseline assessment, the Peru center broke away from a larger parent organization, likely accounting for its unusually high decline in capacity. The change in center capacity reflected in this chart does not, however, tell us whether the ICB project’s organizational viability interventions were successful. First, because the quality of data collected may not be reliable, and second, because it is difficult to attribute the change to ICB interventions. Data quality issues: Dates for the initial and most recent assessments varied with a gap of two to seven years in between each. In some cases, there was a significant time gap between the most recent available assessment and the center’s “graduation” from the project. For example, the most recent available assessments for the center in Bulgaria, India, Pakistan, Peru, and Romania took place in 2003. Bulgaria, India, Peru, and Romania did not graduate from the ICB project until 2006 and Pakistan is still receiving ICB support. As a result, the most recent available assessment may not reflect current reality. As described in section 4.2.1, the way in which the assessment matrices were completed varied from center to center and from baseline to most recent assessment. For twelve centers, CVT recreated baseline stages of development based on notes from assessment visits. Three centers Governance Opera ions & Management Systems Human Resources Financial Resources Service Delivery Information Technology External Relations Initial Most Recent 0 1 2 3 4 5 6 7 Change in Average Organizational Capacity of Centers Over Time 13 appear to have conducted their own baseline without a CVT facilitator. All centers completed the follow-on assessments themselves without the benefit of a CVT facilitator to ensure a consistent approach. Interviews with IMLU Kenya staff indicate that the matrix was completed by one or two senior managers without broader consultation among the staff. By contrast, ACTV Uganda’s former chief executive officer described a participatory process in which the entire senior management team reviewed the matrix and were encouraged to share it with their staff, and in which matrix results were shared and discussed with the board of directors. While both processes resulted in a completed matrix, the later is more likely to reflect broad consensus within the organization and to result in additional benefits. ACTV’s former chief executive officer indicated that the matrix helped the board to recognize actions it should take to strengthen organizational governance. The board reportedly increased and diversified membership and allowed more time for board meetings as a result. In a few cases, CVT questioned the results obtained when centers completed the matrix themselves. In October 2003, CVT asked the then twelve partner centers to complete the matrix. At the same time, CVT completed its own matrix for each center based on its existing knowledge of each center’s capacity. For the most part, the center and CVT assessments correspond. However, CVT and center assessments diverged in at least one Sub-Functional area for nine of the centers. In 60 percent of those cases, CVT rated the center higher than the center rated itself. CVT did not repeat that parallel CVT/center assessment in later years. Difficulty in Attributing Change: Despite questions about data quality, it is clear that some, if not most, centers have increased their organizational capacity during the course of the project. ACTV in Uganda is an example. When CVT began supporting ACTV in 2001, it was an organization in danger of closing. According to an April 2001 CVT trip report, after ACTV’s sole funder stopped supporting the center two years prior, the center dwindled to three staff working on a volunteer basis from the executive director’s home. They had no computers and had served only thirty clients over the previous fifteen months. Interviews with ACTV staff confirm this. Today, ACTV has nearly twenty full-time paid staff, computerized financial and client tracking systems, more than ten funders, annual revenue of more than US $450,000, and in 2008 served more than 2,700 clients.5 Not surprisingly, a comparison of ACVT’s 2001 and 2008 assessments shows significant increases in capacity in virtually all Sub-Functional areas, in several cases improving from “Emerging” to “Sustaining.” A review of CVT records shows, and interviews with ACTV staff indicate that CVT support was in part responsible for this improvement, but that it was not alone in supporting change within ACTV. CVT has provided ACTV nearly US $140,000 in funding since January 2002, which ACTV has used to (among other things) move into permanent office space; support staff salaries; purchase equipment; produce an organizational action plan; and train ACTV staff on project management, proposal preparation, management and team building skills, advocacy and legal issues, database development, and training skills. In short, many of the skills building CVT supported related directly to Sub-Functional Areas in which ACTV has improved. However, comparing CVT support to assessment results does not capture support from other sources that may influence assessment scores. In the case of ACTV, funding from the 5 Information on funders, revenue, and clients served taken from ACTV’s 2008 annual report. 14 Royal Netherlands Embassy supported a full-time international organizational development expert who served as ACTV’s acting director from 2002 to 2003. That expert guided ACTV through a re-organization and helped to hire the chief executive officer, who subsequently oversaw much of ACTV’s growth and capacity building. The flexible nature of CVT’s approach also complicates attribution. Rather than impacting a few key Sub-Functional Areas, CVT’s grants may touch upon many Sub-Functional Areas making it more difficult to attribute change to CVT support. One area CVT has focused on is improving IT capacity via grants to purchase computers and other equipment, developing databases, and on￾site technical assistance. As noted above, that is the area in which average center capacity increased the most. Finally, the way in which CVT captures program reporting makes it time consuming to correlate ICB interventions to improvements captured via the matrix. Centers provide quarterly reports to CVT, but it appears they are not required to include impact indicators that could connect the grants with changes in center capacity. Because centers do not provide cumulative reports on the grants they receive, comparing correlating capacity building grants to changes in the organizational matrix is a time consuming process potentially involving a review of multiple quarterly reports. 4.3.3 Anecdotal Information Regarding CVT Support and Strengthened Center Capacity The best indication that CVT support has positively contributed to organizational viability comes from anecdotal information. According to ACTV and IMLU staff, CVT’s support has played an important role in strengthening their centers. As one individual explained during an interview, ACTV was “like a person on his death bed” when CVT support began. He indicated that in addition to the initial modest grant money, CVT’s support raised morale among the staff and provided stability that allowed ACTV to attract additional donors. According to IMLU’s outgoing executive director, CVT grant funds allowed the organization to significantly improve its Information Technology (IT) capacity. In 2001, IMLU had two computers, only one of which had internet access. Emails were checked once a day, printed, and circulated as hard copies. CVT grants allowed IMLU to purchase computers and upgrade its IT systems. Today, its staff includes an IT and research officer, and most staff have computer and email access. Nonetheless, raising funds to replace aging equipment remains a challenge. According to the IT and research officer, CVT input and funding also helped IMLU to develop a new database with which the organization can more effectively track and serve clients. During an interview, an IMLU staff member said that tips learned during the Resource Raising workshop financed through a grant from CVT helped IMLU to improve its donor outreach, possibly resulting in increased funding, and to form a new advocacy partnership with a Kenyan organization. Both ACTV and IMLU counselors cited new techniques they learned during ICB workshops or regional exchanges and said they were now using those techniques to better serve clients. A 2007 survey conducted by CVT provides additional anecdotal evidence. The center in Ethiopia wrote that attending the ICB workshop in Liberia exposed it to IMLU’s use of advocacy journalism and the Rwanda Center’s use of school clubs to raise awareness. The Ethiopia center 15 claimed to be using aspects of those approaches in its current projects. The Namibia center indicated that CVT support had allowed staff to improve their skills in bookkeeping, trauma intervention, and art therapy, and that the Liberia meeting had exposed the center to useful skills, such as tactical mapping. 4.4 Ideal Size and Resource Allocation per Organization The Scope of Work for this evaluation asks the following questions: • “What is the right size of an organization for the ICB project to be working with? • What is a good balance between number of partners and amount of resources for each partner?” This evaluator suggests reframing these questions to consider not only an organization’s size but its capacity. CVT staff found that weaker organizations lacked the capacity to quickly integrate new learning and also required resources—both funding and staff/consultant time—in order to build capacity. Weak and poorly funded organizations were more likely to use grants to fund core operating costs rather than to channel grant funds into activities that could prove more sustainable, such as professional development opportunities. Size does not necessarily correspond to capacity. CVT notes, for example, that the center in Namibia is relatively small but effective. Determining the size or capacity of centers the project should support and the number of partners it should support depends upon a more basic question: “What is the project’s ultimate goal?” If the project, for example, aims at building strong regional centers that will serve as respected leaders in the fields of clinical services, advocacy, and research for their own and neighboring countries, the project may be most effective if it supports a small group of stronger organizations ready to make the leap from national to regional resource. If, on the other hand, the project hopes to increase clinical services available to underserved populations, it be most effective to support a larger number of organizations (which may be smaller and weaker) that have the potential to increase the number of clients served and extend operations into geographic areas where clinical services are not currently offered. USAID and implementers should also consider the length of the grant and whether the organization has sufficient initial capacity to progress during the life of the project. 4.5 Graduation Beginning in 2006, CVT began to “graduate” centers from the center strengthening component of the ICB project. Five centers (Bulgaria, India, Peru, Romania, and South Africa) graduated in 2006, and two centers (Kenya and Ethiopia) graduated in 20086 6 In addition three centers – Cambodia LICADHO, Nepal, and Rwanda – left the project at their own, USAID’s, or CVT’s direction. . Graduated centers are no longer 16 eligible for CVT grants, but are invited to workshops (such as the 2008 ICB workshop on trauma healing held in Italy) and may host regional exchanges (South Africa has hosted exchanges since graduating). The center in Peru continues as a partner in the IRPEC component of the overall project. To determine which centers would “graduate,” a committee of CVT staff scored each center on five categories: • Level of project participation – whether the center had taken advantage of sub-grants, staff exchanges, trainings, and workshops • Project compliance – including whether the partner provided timely reporting and submitted grant requests based on its Technical Assistance Plan • Duration of Involvement – whether the partner had been supported by the project long enough to demonstrate change/improvement • Leadership – whether the center had the capacity for regional leadership • Location – whether the country or region in which the center was located had a substantial population with current and/or relevant needs Centers scoring high (16 or 17 points out of a possible 23) were, for the most part, “graduated up.” Centers scoring low (9-13 points) were “graduated out.” Centers scoring 14 or 15 points, for the most part, remain with the project. Despite the scoring system, or perhaps because the scoring was closely bunched, CVT’s final decision on graduation was somewhat subjective. For example, IMLU in Kenya, which scored 15 points, graduated while the centers in Palestine (17 points) and Pakistan (16 points) remain ICB participants. CVT staff cited the following considerations in graduating centers: • South Africa (17 points) – Strong and less likely to benefit from CVT support. Client base is increasingly victims of organized violence rather than victims of torture • Peru (16 points) – Strong and financially viable • Kenya – Core activities are medical/legal. Clinical services have not expanded as hoped • Bulgaria and Romania (13 points each) – Both centers scored high for participation and compliance, but low on location. They serve an aging client base and are likely to receive continued funding from European sources • Ethiopia (11 points) – Changes in Ethiopia’s NGO law and restrictions imposed by the government limit the center’s ability to participate in the project • India (9 points) – Did not actively participate in project CVT took steps to prepare for the graduation process. Beginning in 2004, it reduced the amount of funding provided to stronger organizations. CVT discussed the upcoming graduation in its ICB newsletter and, once decisions were made, notified the effected centers. In their responses to CVT’s 2007 survey, the Peru and India centers indicated that the criteria used to select centers for graduation was not clear. The Peru, Kenya, and South Africa centers have each expressed interest in serving as resources for less-capacitated centers. 17 The graduation of centers is a worthwhile strategy for concentrating project resources where they are most needed. The process of determining when an organization is ready for graduation is not entirely clear to centers and the term “graduate” is used loosely within CVT. Graduated centers may have opportunities for continued engagement with the ICB Project via IRPEC, conferences, or as regional exchange hosts, and they may partner with CVT via new projects such as CVT’s new European Union-funded project. 4.6 Human Rights/ Health Care Split This evaluation did not yield much insight into whether it is better to develop the clinical side of a human rights/torture treatment organization or add torture treatment/human rights to a general mental health care organization. According to one IMLU staff member the organization’s Board of Directors has been cautious about adding clinical services to the organization’s mandate. She compared the situation to the board’s initial resistance to adding legal services to IMLU’s offerings. She recommended diversifying the board to include a clinical expert so that fellow board members would be more comfortable expanding that aspect of IMLU’s services. In considering whether to support human rights/torture treatment organizations or general mental health care organizations, USAID and future implementers may want to factor willingness of board members to adopt new activities into their considerations. 4.7 Global versus Regional Focus The evaluation showed pros and cons to continuing a global focus or concentrating on one region in a future project. Overall, there seem to be more advantages in focusing on a single region rather than continuing a global focus. There are advantages to a global project. Feedback that CVT has obtained from partners indicates that there is power in bringing together torture treatment advocates from across continents. According to CVT staff, torture treatment advocates are often isolated and therefore benefit from knowing that they are part of a global movement. Interviews with IMLU and ACTV staff who had participated in exchange visits to the Romanian center indicate that they benefited from the exposure and learned about new models for providing counseling services to victims. One participant later requested and obtained (non-CVT) funding for a regional exchange to Germany based on her positive experience in Romania. She stated that because counseling is a new field in Kenya and the region, it is beneficial to travel to regions where counseling is more developed and participants can interact with experts that have practical experience in counseling torture victims. Retaining a global focus to the project would also complement regional workshops already offered by the International Rehabilitation Council for Torture Victims (IRCT). With the exception of the centers in Guatemala and Namibia, all ICB’s current and past partners are also members of the IRCT network7 7 According to the membership list on IRCTV’s website, www.irct.org and may have opportunities to participate in IRCT regional gatherings. 18 There are however several advantages to focusing on one region rather than attempting to spread resources globally. First, when asked about the project activities they found most beneficial, few IMLU and ACTV staff cited inherently global activities. The conference in Bellagio, Italy, received good reviews. But, the positive nature of the reviews was due to the quality of the training rather than to the opportunity to interact with non-African centers. Several staff mentioned regional exchanges to South Africa, Kenya, and Rwanda as useful and recommended continuing such activities. Clinical staff were more likely to see benefit in global events and exchanges. Management and finance/administrative staff recommended regional events and exchanges, or even sub-regional (for example, East Africa). Second, including participants from multiple regions has occasionally proven challenging. The Peru and Guatemala centers did not participate in global workshops held in Rwanda and Nepal in 2003 due to language barriers. That said, language barriers also pose challenges within regions. Third, a regional project would allow the implementer to develop materials adapted to regional needs and strengthen connections between organizations operating in similar contexts. ICB’s East African partners currently collaborate to implement a European Union-funded project (separate from the new CVT project funded by the European Union). It would be difficult for partners from separate continents to collaborate in this way. Finally, the logistics of conducting a regional project are less complicated than those of a global project. A regional project would allow the implementer to combine site visits more easily and identify one or more regionally based experts who might serve the project, much as Craig Higson Smith serves effectively as a regional resource for African centers. This would allow the implementer to more effectively monitor the project and provide more frequent technical assistance to centers. It may also be more cost effective. 4.8 Holistic versus Focused Capacity Building The evaluation was asked to address the question of whether the ICB project should keep a holistic focus (organizational development, clinical programs, advocacy, and technology) or continue more focused capacity building. The evaluation found several reasons for maintaining a holistic approach. First, organizational capacity in areas such as financial management, governance, and information technology directly relates to an organization’s ability to provide counseling and other services to torture victims. A review of CVT’s 2001 assessment report regarding ACTV, as well as interviews with ACTV staff, indicate that weak management and financial management systems likely were responsible for ACTV losing its sole donor in 1999. As described in section 4.3.2, ACTV reduced the number of clients it served and nearly closed after losing that donor. Even under less extreme circumstances, strengthening financial management, governance, human resources, information technology, and external relations allows an organization to improve performance across the board. For example, introducing sound financial management systems overseen by qualified staff has helped both IMLU and ACTV to diversify and increase their funding bases. ACTV annual reports issued since 2005 show an increase in revenue each year and a 50 percent increase in the number of funders over the period. IMLU annual reports for the past two years also show a 13 percent increase in funding and more than 10 separate funders. IMLU’s client tracking database and the database ACTV is currently finalizing allows those 19 organizations to more effectively and efficiently serve clients. The results are reflected in the services those organizations offer. Between 2005 and 2008, the number of new clients ACTV serves each year has nearly doubled. Second, advocacy can complement and support services to torture victims. As previously defined, advocacy can include policy advocacy, awareness raising, and legal advocacy. Interviews with IMLU and ACTV showed that staff at those organizations saw advocacy complementing other services by improving the country’s operating environment, reducing stigma associated with being a victim of torture, reducing the incidence of torture, and providing redress for those who have been tortured. Third, including organizational strengthening, advocacy capacity building, and clinical services within one project ensures that centers supported will have access to all three. For example, most of the organizational capacity building support CVT has provided could easily come from an organization that does not specialize in torture treatment. The organizational development issues addressed are, for the most part, not specific to torture treatment centers. However, housing these three elements within the same project allows the center to grow in a balanced way and allows the implementer to direct resources where they are most needed. Were USAID to separate these elements out, it likely could not guarantee that the centers would continue to receive organizational development and advocacy strengthening via other initiatives. 20 5. Recommendations The recommendations below are primarily intended for USAID as it prepares solicitations, and for CVT as it considers new project designs. Where feasible, CVT may wish to adopt recommendations for the remainder of its current grant. 1. Improve project reporting by requiring annual reports to USAID. As noted under evaluation limitations, one of the challenges in evaluating this project was the lack of cumulative reporting. At a minimum, the implementer should prepare annual reports to USAID which include cumulative reporting against indicators. Such reporting would improve USAID and external evaluators’ ability to understand key project developments and would likely improve the implementer’s own ability to track progress. 2. Develop a comprehensive Monitoring and Evaluation (M&E) plan with clear outcome indicators and targets. The ICB project lacks a clear monitoring and evaluation plan. The lack of an M&E plan makes it impossible to determine project impact. Future projects should include resources for M&E, and should finalize an M&E plan within the first four months of the project. The M&E plan should include impact indicators, and should measure not only outputs but outcomes. The inclusion of these elements would allow the project to collect relevant baseline information at the outset of the project. CVT’s Organizational Self-assessment Matrix has the potential to capture baseline information if used consistently and if repeated, at a minimum, at the end of the project to capture change. Depending upon the focus of a new project, the implementer may instead wish to use indicators that capture impact related to one or two areas upon which organizational strengthening is expected to concentrate. Examples might include number of new donors over the life of the project, percentage of annual staff turnover, and percentage of clients tracked through database system. The implementer should, at a minimum, include clear cumulative reports on indicators with its annual narrative reports to USAID. 3. Revise the Organizational Self-Assessment Matrix and apply consistently The Organizational Self-Assessment Matrix is a valuable tool. If CVT implements future projects of this nature it should revise the matrix to better capture capacity related to Information Technology and Advocacy. Because client tracking databases have been an ICB focus, CVT may want to include development and use of that type of database as a separate category within the Information Technology Functional Area. CVT should also apply the tool consistently to capture comparable baseline and end-of-project capacity information. If feasible depending upon project funding and length, CVT should apply the tool at mid-term as well in order to track progress and inform decisions regarding graduation (see recommendation 4). The matrix should be conducted by a facilitator so that it is consistently applied across organizations. When implementing the matrix at the mid-term or end of the project, the facilitator’s role could also include leading a discussion about why change occurred, in an effort to determine whether the change can 21 be attributed to the project. Facilitators could come from among CVT staff and trusted consultants, but could also be drawn from regional experts already experienced with similar tools. 4. Clarify graduation process and provide benchmarks for graduation Future projects should plan for graduation from project outset. Such planning would include developing clear expectations with partners about the graduation process, developing a low set of benchmarks that centers are expected to reach to remain with the project, as well as a higher set of benchmarks that indicate a center is ready to graduate, and developing follow-on plans for continued interaction with graduated centers through other project components or non-USAID opportunities. As part of the graduation process, the implementer should assess the center’s strengths and identify areas in which it may serve as a resource organization for other centers. Graduation can be seen as a positive step if it is presented as achieving a milestone rather than as merely losing funding. For this reason, the implementer should use another term to describe those organizations that are “graduated out” due to, for example, lack of participation. 5. Identify regionally based resources for capacity building Regardless of whether future projects maintain a global focus or concentrate on a single region, it would be beneficial to identify regional resources that can provide technical assistance and training on key topics and serve as hosts for regional exchanges. These could include individuals (such as Craig Higson Smith) or organizations (including graduated centers, as appropriate). CVT has already made efforts to do so. Based on positive feedback IMLU and ACTV provided about the assistance Craig Higson Smith provides to African centers, CVT’s efforts have succeeded in this region. Identifying additional regional resources would allow future projects to increase technical assistance to centers, include new centers without spreading existing regional resources too thin, and extend the regional resource model beyond sub-Saharan Africa. 6. Maintain a holistic approach A center’s ability to serve clients is influenced by the quality of its management, financial management, human resources, and information technology systems and practices. Advocacy can complement clinical services. A comprehensive project to support centers should therefore include organizational development, advocacy, clinical, and information technology components. However, a holistic approach runs the risk of spreading resources too thin across project components. To avoid this, future projects may want to target the type of assistance it provides. Projects could develop a shortlist of key competencies centers should have in order to be effective and sustainable. Those could include, for example, an independent board with given skill sets or financial management systems that allows centers to comply with multiple donor requirements and provide timely and accurate financial reports. Future projects could assess centers against these competencies, develop a technical assistance plan with centers aimed at addressing gaps, and gear the level of organizational development and advocacy assistance to meet the gaps identified. Under this approach, centers would not receive support in all areas. Some 22 stronger centers, for example, might receive ICB support to improve clinical and advocacy skills but not for other types of organizational development assistance. A Final Note This evaluation would not have been possible without the efforts of staff from CVT, ACTV, and IMLU who put aside their day-to-day work to educate me on the project. I would particularly like to thank Pamela Kriege Santoso and My Lo from CVT for devoting three full business days and part of their weekend to the Minneapolis site visit, tracking down numerous documents, helping with arrangements for the Africa site visits, and responding to many questions. 23 PART II: CVT’S TARGETED CAPACITY-BUILDING PROGRAMS 6. Introduction Part II of this report concerns the targeted capacity-building programs of the Center for Victims of Torture’s International Capacity Building (ICB) project, the Trauma Healing Initiative (THI), and the International Research and Program Evaluation Collaborative (IRPEC). It was written in conjunction with Part I of the report on the ICB program in general, submitted to USAID by Jennifer Whatley. THI and IRPEC are ICB projects targeted at building (1) clinical intervention and (2) research and program evaluation capacity (respectively) of organizations that address the needs of survivors of torture and political violence in general. This report is a late-term evaluation, required by USAID under its agreement with the Center for Victims of Torture, to examine capacity-building strategies, give some perspective on lessons learned, and provide insight to several broader questions concerning each strategy. As per the stated scope of work of the author and clarification by USAID staff, this report is not meant to be a detailed summary of compliance and administration of the programs since their inception nor a detailed review of quarterly reports, but rather a description of the models, a review of the results of the implementation of those models, and identification of challenges faced in the process and promising activities for similar efforts in the future. This part of the report is organized into four sections: an introduction, a report on THI, a report on IRPEC, and a section on issues that are relevant to both programs. The latter provides space for issues that are not specific to either THI or IRPEC but cut across both (e.g., whether the primary focus of participating organizations should be human rights or health care). Each section is comprised of subsections for specific issues (with the exception of this introduction), with the final subsection being a series of recommendations for similar efforts. Although ICB will be brought to a close in the near future, recommendations are written as suggestions for improvement of current programs in order (1) to be specific about the author’s opinions on challenges and promising strategies, and (2) to comply with USAID’s stated request for specific suggestions for the components of future requests for proposals. The sources of information consulted for this report include the many documents provided by ICB and USAID staff, hours of discussion with ICB staff on the telephone and in their offices at the Center for Victims of Torture in Minneapolis and St. Paul, Minnesota, and conversations and document review during field visits to the Transcultural Psychiatry Organization, Cambodia (TPO Cambodia) in Phnom Penh, Cambodia (which hosts both THI and IRPEC), and the Centro de Atención Psicosocial in Lima, Peru (hereafter referred to as “CAPS Peru” so as not to confuse it with a program in Sierra Leone participating in IRPEC using the same acronym). Findings are based on those sources. 8 8 Opinions are based on the author’s interpretation of findings based on his reading of relevant academic literature (which includes clinical psychology, psychiatry, development, and program evaluation) and prior practical experience observing and evaluating psychosocial efforts in the field. 24 7. Trauma Healing Initiative (THI) The Trauma Healing Initiative (THI) is a capacity building program within the larger ICB structure targeted at building the clinical skills of participating mental health practitioners and increasing the capacity of participating organizations to do clinical training. THI has been implemented solely at Transcultural Psychiatry Organization, Cambodia (TPO Cambodia), a nongovernmental organization with a long history of providing mental health services to Cambodians. According to a number of documents reviewed for this report, TPO was chosen as implementing partner for THI because of its considerable history in mental health, reputation for good governance, and a concordance of goals vis-à-vis training. 7.1 THI Model Description The THI model consists of training practitioners and supporting them through financial support for training time and fostering supportive relationships among them. Its objectives are as follows: 1. Training of trainers 2. Community outreach and education strategies 3. Training the network 4. Network model development9 The scope of THI, according to the comparison chart of CVT’s International Programs (comparing CVT’s direct services, ICB, and THI), is summarized thusly: “Potential to reach hundreds of medical and counseling staff thru [sic] training of trainer model and integration with department of psychology and government psychiatry training programs. Dependent on will and relevance perceived by beneficiaries of training. “Designed for post-conflict country conditions with a high prevalence of trauma-related mental health problems. “Focused intensively on only building capacity in one country among multiple organizations, rather than less intense assistance to single NGO in multiple countries.” The balance of these documents describes a clinical training model for practitioners who treat trauma survivors that is designed to present material on state-of-the-art treatment approaches for post-traumatic stress disorder and general counseling skills, in the context of a supportive network of nongovernmental humanitarian aid organizations, mental health clinics, and related educational institutions. THI is designed to be carried out over four years, and to progress from an introduction to basic counseling skills-like reframing and building rapport, to specific 9 These are consistent with sub-objectives listed in the “Planning Matrix: July 2008 – June 2009” from “International Capacity-building Project: Higher Level Objectives 2,” with the exception of the final goal, which is clarified as “Sub-objective D: Participatory approach to Model development & evaluation.” Other documents reviewed are generally consistent. 25 cognitive-behavioral therapy (CBT) techniques, the place of counseling in non-Western societies, and, in the final year, training others in affiliated organizations in counseling skills. The latter is a key goal of THI, as it is through training that THI proposes to reach as many in the host population as it does. As of this report, THI has begun this final phase of the program. THI’s structure is presented in Figure 7.1 as concentric circles. THI staff aided by affiliated staff at the implementing organization comprise the innermost circle. THI staff train THI “core group members” who represent organizations that work with potentially traumatized individuals. Core group members then become trainers themselves, training their organizations’ other staff, represented by the outermost circle, in a “second-level” training component. Figure 7.1: THI Training Model 7.2 THI’s Target Population THI’s target patient population is broad. Although clearly supported through funding from Victims of Torture legislation, THI’s target population is much larger than torture survivors alone. THI core group members come from organizations that serve populations as varied as human trafficking victims, sexual assault victims, abused children, domestic violence survivors, recent torture survivors, and survivors of the Khmer Rouge regime. This population falls under a larger rubric of trauma survivors, but it would not be accurate to include all trauma survivors in THI’s target population (for example, motor vehicle accident survivors are not represented). Although not made explicit in discussions and documents reviewed for this report, it seems that the target population could be specified as individuals who have experienced personal assaults and are subject to stressors that may be collateral to that assault. Given that THI’s training targets broader academic and skills rather than treatment for specific interventions (although some content appears to have been geared toward working with trauma survivors), the broad definition of the population may not be a problem. In other words, it is member organizations’ other staff core group members THI 26 likely that the approaches and techniques taught to core group members are useful in working with the variety of multiply-stressed individuals they encounter. It does not appear to be the case that any material has been seen as useless to core group members. However, the lack of specificity in terms of exactly what types of clients THI is meant to serve may be a barrier in transferring such models to new settings. 7.3 Providing Broader Academic and Skills Training versus Training to a Specific Trauma Intervention Although taught in the context of trauma-affected populations, the skills taught by THI are for the most part generalizeable to many common mental disorders. Trauma intervention skills taught by THI can be used to treat other disorders as well; for example, systematic desensitization with response prevention (taught in March 2009) is an empirically supported treatment for specific phobias, as well as post-traumatic stress disorder. “Trauma treatment skills” seem to have been presented in the broader context of CBT theory in order to deepen specific intervention skills. As CBT theory is not specific to trauma treatment, it should also provide a basis for applying the skills in related counseling situations. THI trainers have been conscientious about evaluating participants’ knowledge base, and have gone even further to perform observational evaluations of participants’ skills (a rare step in international psychosocial training). Not surprisingly, there is a wide variety of skill levels reflected in these evaluations, with those who have prior psychiatric and psychological training displaying greater skill, and those without displaying less (with one or two exceptions). Ultimately clinical capacity-building comes down to (1) whether skills learned are beyond those that existed before, and (2) whether they are applied properly in practice. From speaking with THI core members, it seems clear that for most, the answer to (1) is an unequivocal yes. For (2), no follow-up system seems to be in place (to be clear, none was proposed, so this is not a failing on the part of the THI project). A couple of accommodations are in place that may facilitate core group members using new skills appropriately. THI has provided an expatriate counseling professional to act as supervisor for core group members (in addition to planning and co-facilitating monthly clinical trainings), however the current person in this position, Ambreen Mirza, reported that this role was more "supportive" than "supervisory." She reported that she provided support on a requested basis to core group members on case management, treatment planning, as well as training. Since assuming her position in late 2008, five of the eight organizations currently participating have requested support (Handicap International, World Vision, and NYEMO, once; Acting for Women in Distressing Situations (AFESIP), twice; and the National Program for Mental Health, three times). A regular (voluntary) Saturday peer supervision may also serve to encourage the appropriate application of new skills. Ms Mirza reported that she regularly provided support at those clinical case discussions as well. A couple core group members gave some indication of which skills they use in practice. Counselor and supervisor Sok Paneth reported that although she herself enjoyed the CBT training, she faced considerable cultural barriers to using it frequently. Challenging and questioning (core techniques in clinical training) are anathemas to Khmer society and can be 27 seen as somewhat disrespectful, particularly when counseling elders. Phaneth reported that she used reflection relatively frequently, and other CBT techniques sparingly. She reported that when she used questioning, she made it clear that she would be asking questions and then reflected by asking about how the clients felt about that before proceeding. Relaxation techniques, she reported, were usually much easier to work with, as several were similar to those Khmer use in entering meditation states in Buddhist practice. As for therapeutic homework assignments (a key piece of many CBT treatments), Paneth reported that about half of her clients—usually those with more education—completed assignments. Dr. Keo Sothy, a TPO psychiatrist, reported that he had begun integrating certain counseling skills into his practice for certain patients. He found that those patients without particularly severe symptoms—those with milder depression and anxiety—were generally good candidates. Still, he reported that the treatment of choice for most of his patients remained medication. He explained that this was because of a preference for medication on the part of Khmer patients in general, and because most patients were from outside of Phnom Penh and thus he could see them no more than once or twice a month (making regular counseling unfeasible). 7.4 Training the Trainers Who Then Train the Network THI Objectives A and C consist of “Training of trainers” and “Training the network.” THI purports to not only train those involved in lessons, but to train staff of participating organizations by having core group members train them (see the second ring in Figure 2.1). Clearly THI staff has made this a central piece of their efforts, and the notion that when one trains one person and then trains them to train others one multiplies the impact of training is a popular notion in the development field. THI also sees training others as a central learning experience as well, because in teaching one is re-exposed to the material in explaining it to others. As the “Training the network” phase of THI had only begun during the writing if this report, it is not yet clear how those efforts will proceed in practice. THI has put in place a few conditions to increase the likelihood that core group members will train others in their organizations. First, that organizations would have their representative train their own staff was a condition of participation in the THI core group. Second, THI staff have been in contact with network organization administrators to remind them that one of the conditions of participation is that their representative train others in the organization (e.g., during the June 2009 dinner meeting of the heads of organizations, hosted by THI at a restaurant in Phnom Penh). Third, THI has a system in place to have core group members report to the group when they train others in their organizations. Fourth, THI is developing an interactive audiovisual manual of the skills that THI has covered, which it will distribute to all core group members once it is complete. There are a number of challenges to the goal of training the network effectively. First and foremost is the variety of skill levels within the group, both clinical and pedagogical. The core group includes psychiatrists who have taught many courses at the Royal University of Phnom Penh, and social workers whose only exposure to mental health has been through THI. Second, some organizations have training as their mission (e.g., the university) and others hire individuals precisely for their knowledge (and therefore in order to avoid training anyone). Third, the 28 migration of some staff from member organizations to TPO (see below in section 7.7) has reduced the capacity to reach a wider network. How substantial those challenges are cannot yet be assessed, but it is likely that there will be some loss of quality in the transfer of skills from core group members to their organizations’ other staff. In Figure 2.1, this is represented by the difference in tone between the second and outermost circle. By THI’s end (June 2010), THI staff should be able to report how many organizations have implemented those organizational trainings. It is not clear that there is a mechanism in place to evaluate the quality of the trainings or the use of skills trained by those outside of the core group members. 7.5 The Role of Manualization in Sustaining and Disseminating Clinical Training It is unclear what role the interactive audiovisual manual will have in sustaining and disseminating clinical training. The manual at this point is not yet fully developed, nor has anyone been trained using initial drafts. Given the minimal number of comprehensive resources in psychology in Khmer, it is likely that the manual will have an influence once fully translated and finalized. Certainly, the development of the manual has become critical to THI’s efforts in the fourth year. The manual is designed to be used as a reference and to help core group members train others outside of the core group. Given that the “train the trainers” course consisted of five days of training, and only half of one day was devoted to reviewing a draft version of the manual, it seems unlikely that those whose skills have come entirely from THI will go on to use the manual in training. However, those with more experience teaching (for example, those involved in medical education) may find the manual quite helpful. THI’s sub-objective A5 mentions the development of “modular culturally appropriate treatment and training manual” (p. 2, “International Capacity-building Project: Higher Level Objectives 2”). It is not clear what is meant here by “culturally appropriate.” The manual in development seems to refer mostly to standard Western treatment modalities. Adaptation has involved translating English text on psychological constructs into Khmer and using Cambodian examples. THI has included training in how Buddhist concepts have been integrated into counseling practice (which is included in the manual), and it is a sign of the potential of integrating culturally-based practices that this was one of the first things mentioned by core group members when asked about the highlights of their THI experience. However, nowhere in THI literature reviewed is there reference to the clinical literature specific to Cambodians—of which a fair amount exists (relative to literature specific to other ethnocultural groups). There are validated treatments, with training manuals, and a host of literature on idioms of distress among Cambodian trauma survivors. To be clear, this is not to say that THI material is somehow harmfully inappropriate (the somewhat badly-coined term “culturally-appropriate” does not automatically imply that using non-culturally-specific techniques are therefore inappropriate). More discussion on the literature on psychological distress among Cambodians does, however, seem warranted. On a related note, THI’s sub-objective D, “Participatory approach to Model development & evaluation” raises several issues. Clearly, participation creates an investment in the process of development, one that likely increases with increased participation. However, sub-objective D2 29 goes one step further when it suggests that THI “establish a consensus building approach among key clinicians and decision makers for determining best clinical practice.” Who are these key clinicians? Is consensus building the best way to determine “best clinical practice” in a setting in which providers are asking for outside training? It may be that this point is moot: THI staff seem to have decided that certain materials are appropriate for training trauma treatment providers (e.g., CBT), and there does not seem to be documentation of meetings between key clinicians aimed at coming to consensus. It may also be that the language of this sub-objective is vague enough to be interpreted several ways. 7.6 Khmer Rouge Tribunal Project A somewhat ancillary program to THI training but funded in part by the THI project is The Khmer Rouge Tribunal (KRT) project. The KRT project focuses on providing services to tribunal witnesses and civil parties involved in the Extraordinary Chambers in the Courts of Cambodia (ECCC), the trials of the leadership of the Khmer Rouge. This project, although not comprised of direct clinical training, is consistent with THI objective B, “Community outreach & education strategies.” Dr. Sothera, who is a THI core group member, leads the KRT project. He and his team work with witnesses and other KRT participants in several ways. The first element of KRT involves being on call for any courtroom participants through a telephone hotline and providing onsite support when requested. The second involves providing outreach and direct support to the civil parties; KRT staff has done outreach to all ninety-four civil parties named in the trial of Duch. They have met with each through partner organizations, followed up with them by phone after one week, and provided some basic psycho-education and relaxation techniques as preventive measures for the anticipated stress of the tribunal. Following initial assessments, civil parties were prioritized in terms of clinical severity, and those in immediate need of services were given follow-up contact more frequently. The third way that TPO’s KRT project supports KRT participants is through their memorandum of understanding with witness support services at the ECCC itself. TPO staff members are on call when witnesses go on the stand. TPO staff attend the hearing (one in the court, one in the preparation room) in order to prepare witnesses for sudden emotions they may have in court, all the while being very careful not to influence the testimony. Following testimony, TPO staff debrief clients. In addition, TPO staff working on the KRT project is involved in training personnel in organizations working with witnesses. Staff train partner organizations in recognizing trauma reactions and preventing secondary trauma. Through the Khmer Institute for Democracy (KID), they trained the 150 officers who were eligible to work in the KRT during the KID training on protection of witnesses and the defense. KID also trained citizen advisors who help civil parties complete files. TPO trained those 130 individuals also, as they were thought to be candidates for secondary trauma. They also trained the research personnel from the Center for Advanced Study (CAS) in working with trauma survivors during documentation projects. The KRT project represents an impressive breadth of services to witnesses, civil parties, and support staff involved in the ECCC. Although none of the services fall under the intervention skills specifically taught by THI, the KRT is using appropriate psychological concepts and support consistent with THI’s mission. 30 7.7 THI’s Value to Implementing Organizations TPO is overall very positive about THI. Clinicians see THI as valuable to furthering their clients’ treatment aims. TPO administration sees THI as furthering broad capacity building goals, particularly those related to becoming a leader in training others in mental health in Cambodia. Pich Panha, a counselor at TPO, cited the monthly consultant trainings as programmatically important to maintaining the network and improving the training received by core group members. In addition, he reported that THI core group members have taken it upon themselves to organize a peer counseling session twice a month on Saturday mornings in order to share counseling experiences and review topics. This meeting is optional, and evidence of the investment several core group members have made in their training. Mr. Panha attributed that investment to the careful needs assessment THI undertook with partners before the training began in earnest. Sok Paneth, a TPO counseling supervisor, also expressed enthusiasm surrounding the Saturday morning sessions, citing them as a good model for peer support. She reported that core group members would like increased frequency of clinical training opportunities, beyond the monthly training sessions. Director Dr. Sotheara Chimm reported that he sees the future of TPO for the next three to five years as moving from donor-driven funding to income generation. One of the key pieces of income generation is the training of other organizations in mental health concepts and practice. Dr. Sotheara sees THI as integral to improving the knowledge base of his staff, and therefore to building the capacity of TPO in general. He reported that training already brings TPO $30,000 a year, making the claim that THI directly contributes to meaningful organizational expansion all the more feasible. He hopes that international agencies will come to rely on TPO the way many local organizations have; he reported considerable frustration that the local USAID office hired US nationals for $10,000 to $20,000 to come to Cambodia to train staff at the Documentation Center of Cambodia (DC CAM) in basic psychological concepts, when TPO had the skills, was available, and trains in Khmer (not to mention would charge considerably less). An unintended beneficial consequence for TPO has been the transfer of talented staff from other partners in the core group. Three of seventeen core group members at the June 2009 training began as members of partner organizations, were recognized for their talent through THI, and were hired at TPO (one of the members has since left TPO in order to pursue a teaching career). This has undoubtedly been a benefit for TPO and increases THI’s clinical influence within TPO. However, as there has not been subsequent efforts to replace those persons from the same organizations, the transfers have also resulted in the reduction of the THI network overall, and have therefore reduced the reach and impact of THI in Cambodia. Perhaps intended and equally beneficial has been the effect THI has had on TPO’s position as a central player in mental health in Cambodia. At the dinner to discuss THI (see above, in “Training the trainers who then train the network”) were government figures from the health ministry, as well as the heads of partner organizations. Although it is clear that TPO Cambodia was already a major player in mental health in Cambodia before THI began due to its lengthy history work in community mental health in rural provinces, it is also clear that THI has 31 furthered this position and expanded its work in other aspects of mental health delivery (e.g., expanded its outpatient clinic in Phnom Penh). The latter point raises a central question in looking toward the future: What are the organizational characteristics of a potential implementing organization that indicate that it is likely to be successful in receiving training and disseminating this information? The experience of THI and TPO to this point may provide some insight. Two conditions prior to beginning THI may have been important: TPO had a proven history of providing mental health services and TPO had strong relationships with local health providers and with the Royal University of Phnom Penh. Without any comparison organization (i.e., THI has only been implemented once, at TPO) the importance of those factors is only speculation, but it seems reasonable that without one of those factors the capacity to implement and ability to disseminate would have been compromised. Also important may have been TPO’s prior history with the briefer mental health trainings that (unfortunately) characterize the field. The commitment to longer-term relationships between trainers and trainees characteristic of THI may have made TPO more committed to the project given a history of disappointing “here-and-gone” trainings. 7.8 Conclusions, THI THI is an in-depth training protocol of empirically validated trauma treatment techniques for staff members working with multiply stressed populations. Most mental health trainings in international settings consist of brief classroom instruction and are designed without adequate knowledge about potential trainees. By contrast, THI represents a comprehensive, long-term commitment to training and is designed to specifically address those common problems. Strengths include the time invested in initial assessment of training needs, a strong evaluation component based on clinical observation, the relationships built with TPO staff members, the emphasis placed on building relationships between trainees themselves, the production of an interactive electronic manual, and the placement of one expatriate staff member on site. Challenges include the wide variety of counseling skills and levels of knowledge about mental health concepts among trainees, the integration of culturally-specific mental health concepts into training, and follow-up with the application of skills across the wide variety of settings in which core group members work. 7.9 Recommendations, THI The following recommendations pertain to future mental health training programs funded by USAID: 7.9.1 The organizational characteristics of the implementing organization should include 1. a proven history of at least three years of mental health service provision 2. at least three full-time mental health staff members (not funded by the proposed training program) 3. established relationships with local health structures and/or local educational facilities 32 4. a commitment to training those outside of their organization 7.9.2 The target population of the training program should be specified. 7.9.3 THI has provided many aspects that should be included in a list of recommendations. Those measures have placed THI above the norm for international trainings; should be included as pieces of future projects; and should be expanded, particularly evaluation efforts. Those include 1. the comprehensive assessment of training needs 2. clinical observation and ratings following training clinical skills 3. a commitment to a long-term relationship with the implementing organization 7.9.4 Implementing programs should refrain from hiring trainees during the training period in order to maintain the maximum reach of the training network. 7.9.5 If “training the trainers” is to be a key goal of future applications, mechanisms should be put in place to evaluate the quality of the second-level training and monitor the use of skills of those not directly trained by program staff themselves. 33 8. International Research and Program Evaluation Collaboration (IRPEC) The International Research and Program Evaluation Collaboration (IRPEC) is a capacity building program within the larger ICB program, targeted at building research and evaluation infrastructure by transferring the technology of measure development and research design to participating organizations. Originally designed for four organizations, IRPEC has involved six organizations to date: Campaign for Innocent Victims in Conflict (CIVIC) (Nepal); Sach! (Pakistan); IMLU (Kenya); CAPS Peru; TPO Cambodia; and CVT’s own International Program in Sierra Leone, which became its own independent organization, CAPS Sierra Leone. The four ICB organizations originally identified for the IRPEC program were CIVIC, IMLU, CAPS Peru, and CVT/CAPS Sierra Leone. The history of participation is discussed following a description of the IRPEC model. 8.1 IRPEC Model Description and Re-Description IRPEC’s primary goal is stated as “to strengthen research capabilities at ICB centers to enable them to measure the impact of rehabilitation interventions on the functioning and well-being of torture survivors” (page 1, “International Research and Program Evaluation Collaboration (IRPEC): International Capacity Building in Monitoring, Evaluation and Research,” dated October 2004). IRPEC is explicitly conceived of as “a 5-6 year initiative” (Ibid.), with the following seven specific goals (paraphrased here; quotes appear in previously indicated document): 1. increasing monitoring and evaluation capacity 2. building research capacity, particularly for intervention research 3. developing “regional centers of excellence” that will aid other centers 4. creating a “basic” set of relevant “evaluation measures and methodologies” to be used for “advocacy/education and treatment activities” 5. creating a set of “refined research measures and methodologies” to be used to evaluate “effectiveness and impact of torture treatment interventions” 6. conducting research and examining findings on effectiveness of intervention strategies 7. laying “the groundwork” for building “an empirical basis” for measuring the effectiveness of torture treatment interventions across contexts and cultures Holistically, those goals represent an attempt to engage programs in the process of evaluating their work in clinically meaningful ways, and to do so by developing their own local knowledge in measurement design and interpretation of results. In essence, IRPEC aims to be a technology transfer program, in which the technology is comprised of research and evaluation expertise. That approach goes beyond evaluation research, in which outside evaluators bring in their own technology (their expertise, standard Western measures, statistical skills, etc.), report their 34 findings, and leave the organization with the same degree of capacity as when they arrived. IRPEC is an education program in which participants learn evaluation by designing projects as well as carrying them out. IRPEC staff guide organizations in identifying their evaluation needs and in selecting the methods they will need instead of directing them to specific goals and measures. In that way, IRPEC aims to sustain and ultimately foster further development (i.e., development after IRPEC is gone) of evaluation technology outside of North America and Europe. Implicit in the goals above is a division between research and program evaluation. Although many of the same skills are required for both sets of activities, the goals and standards of research and program evaluation are different. IRPEC’s division of research and program evaluation written into those goals appropriately provides for those differences, making it easier to record progress toward one or the other. Consistent with their overarching goal, IRPEC explicitly states that its model focuses on the core values of (1) being inclusive of all staff in building capacity; (2) building collaborative relationships between agencies; (3) focusing on functioning, as well as trauma symptoms; (4) valuing local beliefs about well-being, and including those in measures; and (5) assessing impact beyond the individual. According to the 2004 document referenced above, IRPEC was designed to be implemented according to a detailed timeline consisting of three phases. Phase one involved identifying participating organizations as regional sites according to specific criteria, identifying local expertise in the area, training in basic research skills and measurement selection, developing site￾specific training, and implementing measures. Phase one was to have taken eighteen months. Phase two involved developing participating organizations’ statuses as regional sites, and was comprised of formalizing participating organizations’ own evaluation and research, and identifying “second tier centers” to be supported by the original participants. This phase was to take two years. Phase three involved implementing intervention research at regional sites, beginning second tier sites’ formal program evaluation, and collaboration across regional centers, and was to have taken two years. That model appears in Figure 8.1. Figure 8.1: Original IRPEC Model (2004) Phase 1 Phase 2 Phase 3 Identify & train 4 Regional Centers Program evaluation at 4 Regional Centers Intervention research & collaboration Identify & train 2nd-tier centers Program evaluation at 2nd-tier centers 35 According to Jon Hubbard, in anticipation of the beginning of Phases 2 and 3, USAID requested that the IRPEC model be more focused on gathering research and evaluation data. In the rewritten proposal, IRPEC was scaled back to eliminate second-tier sites and to focus on developing capacity in three sites from which to gather data: CAPS Peru; CVT’s own center in West Africa, which would soon become CAPS Sierra Leone; and a recently identified organization with a reputation as a well-respected mental health program, TPO Cambodia. With this rewritten proposal came a pared-down model of IRPEC, represented in Figure 8.2. Figure 8.2: Rewritten IRPEC Model (2006) Phase 1 Phase 2 Phase 3 This “re-described” model departed from the formal Regional – Second-tier Center structure. However, the concept of developing participating organizations as Regional Centers was not abandoned by IRPEC. Participating organizations remained conceptualized as regional resources, although the sites for which they were resources were no longer to be specified. The conceptual model became, in effect, that represented in Figure 8.3. Figure 8.3: Effective IRPEC Model (2006 to present) Phase 1 Phase 2 Phase 3 Without specified secondary sites, technology transfer throughout a region was left to the resources of local institutions (as opposed to those provided to 2nd-tier sites by the IRPEC grant for such purposes), and their willingness to pay for primary sites’ services. This undoubtedly curtailed the capacity of regional centers to have an impact beyond their own programs. It did not, however, eliminate that capacity. Notably, the goals of IRPEC were not changed. Identify & train 4 Regional Centers Program evaluation at 3 Regional Centers Intervention research & collaboration Identify & train 4 Regional Centers Program evaluation at 3 Regional Centers Intervention research & collaboration 36 A final note about the IRPEC model: Although “torture survivors” appears related to IRPEC goals, the techniques used in IRPEC are not specific to centers that serve that population, and indeed, the target population is ultimately broader than torture survivors alone (similar to the target population for THI; see section 7.2). 8.2 IRPEC Participation History The history of which organizations have been able to participate in IRPEC and which have not is somewhat complicated. Original sites were CIVIC, IMLU, CAPS Peru, and CVT/CAPS Sierra Leone. Shortly after IRPEC began, CIVIC’s participation was discontinued because of increasing armed conflict in Nepal, and Sach! was suggested (by the head of CIVIC) as a regional alternative. Although Sach!’s structure did not seem entirely appropriate to IRPEC staff because of its emphasis on minimizing the involvement of professionals in community mental health work, IRPEC staff felt that they might still be able to encourage the leadership to expand their professional staff. However, the organizational structure proved to be resistant to change, and increasing violence in Pakistan made it readily apparent that Sach! was going to be very difficult to develop. By the time that Phase 1 of the model was ending (at the same time that the IRPEC proposal and model were rewritten), IRPEC had decided that involvement with Sach! was not worth continuing as intended. Another of the original participating organizations, IMLU, was discontinued because the planned mental health component of the organization was not developed as anticipated. IMLU’s focus is documenting human rights abuses, primarily through the use of forensic autopsies. Jon Hubbard reported that IRPEC began its involvement shortly after IMLU decided to add a mental health component to its services (for witnesses) by hiring a number of counselors. Plans were changed shortly after IRPEC began working with IMLU, and mental health resources were limited to one psychologist. In a conversation with Jennifer Whatley on her ICB monitoring trip to IMLU, the IMLU Director informed her that from IMLU’s perspective IRPEC was substantially more directive than the previous ICB involvement, and that this led to the termination of the program. This is not to say that IMLU did not benefit from IRPEC involvement. Dr. Hubbard reported (and IRPEC reports confirm) that IMLU improved the way they collected information about their work and the way that they communicated what they did to those in civil society in Kenya. However, Dr. Hubbard also reported that without a counseling program, it was clear that their program would not be able to take full advantage of IRPEC. By the end of Phase 1, IRPEC’s involvement with IMLU was in the process of being discontinued. Also at that time, IRPEC began its involvement with TPO Cambodia. TPO had already been identified as a reputable organization with impressive capacity, and had by that time been participating in THI for some time. Dr. Soethara Chimm reported that before IRPEC TPO had had several experiences with outside researchers providing research and evaluation services without leaving any increased capacity of the organization when they left (similar to their experience with clinical training). TPO staff had a strong desire to build evaluation into management and clinical work, but had only basic evaluation skills. IRPEC’s emphasis on building local capacity for evaluation therefore fit precisely with TPO’s needs. 37 IRPEC’s work with The Center for Victims of Torture’s own center in Sierra Leone has proceeded somewhat differently from its work with other participating organizations. In order to represent the transition that this organization made from being in The Center for Victims of Torture International Program to becoming an independent organization over the time period that IRPEC began, this report will refer to that organization as CVT/CAPS Sierra Leone (it should not be confused with CAPS Peru, which shares an acronym but no institutional connection). CVT/CAPS Sierra Leone’s evaluation staff initially worked for Dr. Hubbard, (the Director of IRPEC) and therefore the “advisory” orientation of IRPEC was instead more direct. As The Center for Victims of Torture left Sierra Leone (following repatriated refugees to Liberia) and CVT/CAPS Sierra Leone staff planned their own independent organization, they remained committed to the clinical model in which they were trained, in which evaluation and assessment are central components of effective treatment. They therefore remained an IRPEC participant after their institutional break from The Center for Victims of Torture. Dr. Hubbard reported that he sees the maintenance of that value following an organic transition to independence from the parent organization as a clear sign of success. As a sign of the success of the “assessment as clinical practice” model, it most certainly is. However, as the initial stages of IRPEC’s involvement with CVT/CAPS Sierra Leone were institutional, it is less informative than experiences at other sites concerning the potential of IRPEC to influence an organization in an advisory capacity. Central to this evaluation is the issue of program readiness for a targeted program like IRPEC. What needs to be in place at an organization for IRPEC’ objectives to be most effective? The history of organizations participating in IRPEC may be instructive. The history of CIVIC and Sach! suggest that, given IRPEC’s regular training visits, security is a central necessity. Sach!’s priority of relying on community mental health workers and not involving professionals suggests that organizations must be willing to involve new staff with some level of professional training. IMLU’s lack of a sustained mental health program was reason for IRPEC to pull back from full engagement. CAPS Peru and TPO Cambodia’s prior experience in mental health suggest those programs with proven records of mental health service delivery are more appropriate than human rights programs with added mental health components. As important as their record in delivering mental health care may have been the existence of preliminary data infrastructures. Directors at both TPO and CAPS Peru reported that prior to IRPEC they had basic forms for recording service use data (e.g., the dates of clients’ visits and who provided clinical supervision for junior staff working on specific cases), simple databases for doing so, and, in TPO’s case, the use of an evaluation instrument in treatment. With funds from IRPEC those databases were expanded (in the case of CAPS Peru) or recreated entirely (TPO). Essential to IRPEC’s success was the presence of executive directors interested in research. Both Sotheara Chimm (TPO Cambodia) and Carlos Jibaja (CAPS Peru) reported that they had had direct involvement in research themselves prior to IRPEC for purposes of professional development (Dr. Sotheara in pursuit of his PhD at an Australian university; Dr. Jibaja as a member of the local psychoanalytic organization in Lima). Dr. Jibaja stated that he thus made research and program evaluation one of his priorities when he began his directorship, and that this was a shift in focus from that of the past director. Other institutional characteristics that may have been important include historical factors. CAPS Peru had a history of involvement from the local USAID office in encouraging the use quantitative data in program monitoring and evaluation prior to the entrance of IRPEC. Dr. Jibaja 38 reported that the project was actually quite similar to ICB in that it focused on capacity building, and, according to Cecilia Velasco, USAID’s assistant mission director, it was even funded by money from the Victims of Torture Fund. In addition, IRPEC arrived just as CAPS Peru was planning to expand its services to communities outside of Lima, which required improved monitoring. As stated above, TPO Cambodia had a history of involvement in research projects designed and carried out by European researchers. That history was somewhat disappointing in that TPO staff believed that there was little they had learned from the research (as it had been designed, coordinated, and analyzed by foreigners), but it did provide them with an interest and some experience in data collection. 8.3 Data Management and Analysis IRPEC goals (a) and (b) concern increasing evaluation and research capacity. IRPEC quarterly reports seem to indicate that those efforts have proceeded at a steady pace at CAPS Peru, CVT/CAPS Sierra Leone, and TPO Cambodia. In general, those efforts have involved initial visits to assess needs, the provision of training in database software and evaluation design, and regular follow-up supervision and consultation visits. The reports do not indicate any major problems in data management (e.g., significant loss of data or compromised confidentiality). Methodological Sophistication During the visit to TPO Cambodia made for this evaluation, Data Manager Deup Channarith reported that although his prior training in demography (in Belgium) provided him with a base understanding of the methods used in his current work, it has been through his involvement with IRPEC that he has learned how to properly apply those methods. That training has included several on-site visits from IRPEC staff, as well as supervision (using videoconferencing software) two to three times a month with IRPEC staff. Responsibilities at TPO include cleaning data, data management, and some development and design of program evaluation efforts. During the visit to CAPS Peru made for this evaluation, Research Director Carlos Saavedra reported that IRPEC has required that he use statistics skills he had learned in pursuit of his degree in engineering (his undergraduate field of study), as well as qualitative methods he learned in pursuit of his master’s in anthropology. Saavedra reported that although he participated in regular supervision chats and phone calls (using Skype), most of the skills that he used at CAPS Peru were those he had learned prior to his involvement in IRPEC. Saavedra noted that the supervisory calls and chats suffered from technological difficulties and differing priorities of the centers involved, and he believed that less regular, person-to-person supervision would have been more effective. Saavedra cited the visit of The Center for Victims of Torture researcher Greg Vinson as an example of effective training. From Dr. Vinson, he learned about the design and analysis of sorting techniques (used during the development of the IRPEC measure). Carlos Jibaja noted that CAPS’s expectation had been that researchers associated with IRPEC would function at a higher level of methodological sophistication than would CAPS staff, and that the organization was somewhat disappointed that the normal supervisor, Emily Williams, did not have that skill level; he noted that her skills were sufficient to help complete 39 the projects, just not at the level of that would contribute to progressing beyond current capabilities. In order to examine the level of understanding gained through IRPEC, research and evaluation staff at TPO Cambodia and CAPS Peru were asked to explain statistical results included in existing reports. At TPO, Mr. Deup was asked to recreate a histogram depicting change in Blue Cloth assessment scores over the course of treatment featured in an annual report provided by Sam Mara, TPO’s Director of Operations, and then to test the statistical significance of the change. In order to complete the chart, Mr. Deup first calculated the mean scores of each item in SPSS (a statistical software package), pasted those results into Excel, transposed the mean rows into columns, and then ran a histogram. That process demonstrated an in-depth understanding of transferring analyses across statistical packages, but failed to account for missing data between assessment points (i.e., missing data at twelve-week and six-month assessment points were not accounted for in the analyses) and failed to demonstrate understanding of the value of summary scores. Summary scores are usually more reliable indicators of symptom severity than individual symptom ratings, because they are comprised of multiple measures (multiple symptom ratings) of a single clinical phenomenon. When calculating statistical significance of change in symptom severity over the course of treatment, Mr. Deup used t-tests appropriately (which do account for missing data), but again only examined change in individual symptoms, not summary scores. Those errors are common among doctoral students using statistical analyses at universities in the United States. When shown those errors, Mr. Deup seemed to quickly grasp the implications of his errors and the means to avoid them in the future—signs of useful statistical knowledge. At CAPS Peru, Carlos Saavedra guided the author of this report through an extensive draft of his report on the use of quantitative and qualitative data. In contrast to Mr. Deup, Mr. Saavedra seemed facile with many statistical analyses, including scale reliability analyses, paired samples t-tests, Cohen’s d-scores, Wilcoxon Signed Rank test (for non-normative data), and exploratory factor analysis. Mr. Saavedra’s facility with qualitative analysis (gained in his training in anthropology) has provided CAPS Peru with an additional asset that may aid in interpreting the improvement it sees in the first three months of treatment. As of September 2009, Mr. Saavedra is working on methods to present his results to clinical staff members who are not as facile in statistics in order to provide a useful framework for applying those results to clinical work. Mr. Saavedra reported that he and Director Carlos Jibaja plan to publish those results in January 2010 (CAPS Peru has regularly published reports on its interventions, but it has yet to publish research). Using Data to Inform Practice One of IRPEC’s goals is to encourage organizations to use data to inform their practice. Both Mr. Saavedra and Dr. Jibaja reported that one of the most effective uses of data at CAPS Peru was a quick analysis done by Mr. Saavedra at IRPEC’s outset in which he used service utilization records that had been collected for some time prior to IRPEC. The records indicated that many clinicians had the impression that most of their clients attended therapy for long periods of time, and thus were appropriate for the type of in-depth, long-term psychoanalysis that they had been trained to do and that they regularly carried out in their private practices. Mr. Saavedra reported that, according to the records, on average clients only attended seven sessions. The clinicians were surprised by the magnitude of difference between the therapists’ impressions 40 and the records, and consequently reconsidered the therapeutic approach to their practice. Dr. Jibaja proposed that therapists begin using modes of brief therapies (twelve sessions) as an alternative, in which therapists focused on specific problems rather than delving into their patients’ psyches. At a staff meeting during the evaluation visit for this report, several attending therapists commented favorably on this change, noting that CAPS Peru’s clients do not have the luxury of participating in long-term dynamic therapy. Other uses of data to influence practice have not been as successful. Although the scale created for IRPEC has been integrated into the intake packet for new patients and is administered after three months, therapists at CAPS Peru reported that they rarely look at the results when conceptualizing cases. In the CAPS Peru staff meeting, some therapists proposed that the scale was more useful to the clients in that during administration it helped them define their problems for themselves. One therapist (the most recent hire) reported that, at times, she looked at scale scores before seeing clients for initial sessions. Erica Cuba explained that many therapists expressed concerns that examining scale scores (or any information) prior to initial visits contaminated the therapeutic process (this professional bias against information outside of therapy is based in psychoanalytic theory, and not uncommon among analysts). Carlos Jibaja confirmed this, and proposed that an appropriate use of the scale might come after the initial therapeutic evaluation was complete (three to four sessions); he reported that he would propose that approach to the group. Mr. Saavedra reported that many of the therapists were more amenable to feedback based in qualitative investigations. He and the other two members of the IRPEC team, Erica Cuba and Haydee Antón reported that their qualitative study of therapeutic perspectives would be reported to the clinical team in the coming months. That study comprised interviews with therapists and clients involved in therapy concerning the nature of therapy and clinical improvement. Saavedra and his colleagues explained that they are analyzing the data along several “axes” —e.g., interpretations of clinical improvement, motivation for therapy, explanation of the therapeutic process—and comparing those results to results from the IRPEC scale. Preliminary results suggest that despite agreement that clients improve in general, the perspectives on therapeutic process and motivation are quite different. CAPS Peru research staff members hope that discussing those differences might aid in furthering the influence of research on practice. Expatriate Evaluation Experts A central issue in the evaluation of IRPEC concerns the role of expatriate evaluation experts. Would IRPEC be well served by posting a full-time, on-site expert at participating organizations, instead of having staff conduct month-long visits and regular supervision twice a year? Although the addition of a statistically advanced, well-trained free expert would be an obvious temptation to any program director, it is likely that placing an expatriate staff member in a position to oversee evaluation and research at each participating organization would be unnecessarily expensive, and it might even work against the development of sustained capacity in the long term. One of the strengths of IRPEC training and supervision is that it encourages center staff to use the skills and technology transferred to them on their own, guiding them where needed. With the placement of expatriate staff on-site, it is likely that some of this “learning-through￾exploration” would be lost. An expatriate staff who might successfully avoid this situation by 41 using a “hands off” approach to supervision would, in effect, not be much different than IRPEC staff off-site. 8.4 Review of IRPEC Scales IRPEC’s goals (d) and (e) involve creating measures for mental health program evaluation and research. IRPEC assists program staff in developing the methods for scale development (which they label “ethnographic”) and choosing response scales, in addition to the training they provide in data management and analysis. One concrete indicator of those efforts is the quality of the outcome measures that have been developed. What follows is a critical review of each of the four measures developed and used by the organizations involved for historical reasons, stated earlier). CAPS IRPEC Scale The measure resulting from IRPEC work at CAPS Peru is a thirty-eight-item measure with a five-point response scale (from always to never). Items are a mix of psychological symptoms of anxiety (e.g., “Do you become nervous?”), depressive affect (“Do you feel that you are not worthy of much”), and post-traumatic intrusion (“Do you have strong, repeated memories of horrible images that you have lived through?”); maladaptive behaviors (“Have you ever hit or attacked physically other persons (neighbors, friends, family)?”); functioning (“Are you able to complete things that you start?”); and a few global emotional states (“Do you feel good about yourself?”). The measure assumes that respondents are survivors of political violence (items include, “Do you avoid talking about the violent political times?” and “Do you feel that you are able to reclaim your rights?”), as opposed to a general measure focused on psychopathology alone, and includes a number of behaviors oriented toward participation in community activities (“Do you participate in social or community activities?”). This scale covers many of the symptoms, behaviors, and functional problems common to individuals who have experienced significant stressors (including potentially traumatic events). It includes a suicidal behavior item. The scale includes positively worded items, which allows respondents to express their positive emotional experiences in addition to the negative experiences. The orientation toward survivors of political violence is an asset for working with the clients specifically served by CAPS, but may be limiting in the future should the project serve individuals who have not been exposed to those events (a simple modification would suffice to alter the measure to make it more general). Martha Stornailulo, who is responsible for administering the scale to all incoming patients, reported that because the scale was designed using items culled from provincial populations, an additional drawback might be that it does not represent symptoms among the usual patient population at CAPS, who are from Lima. The one major drawback of the measure is the lack of an explicit time period in which to anchor responses. For instance, should respondents answer, “Does your body tremble?” thinking about the moment they are asked, over the past week, past month, or ever in their life? When asked about the timeframe of the measure, Erica Cuba, an anthropologist on the Monitoring and Evaluation team, explained that the timeframe was left unspecified because the scale was intended to measure whether or not symptoms had ever occurred since exposure to civil conflict 42 (referring to the civil conflict with the Shining Path guerillas). Carlos Saavedra reported that the timeframe was left unspecified because it was intended to measure the patients’ current emotional state. Dr. Stornailulo (who administers the scale), reported that in the absence of specific instructions the time period for the measure was left up to the patient. In discussing this issue with Carlos Jibaja, Dr. Jibaja agreed that it would be a good idea to clarify this. During the visit to CAPS Peru, Mr. Saavedra reported that the internal reliability of the measure was high (Cronbach’s α = .91), and that the scale contained eight empirical factors that corresponded roughly to the eight theoretical dimensions that clinicians proposed during item reduction. That analysis was described as preliminary, as data collection is ongoing. IMLU Assessment That assessment is designed to be administered over two counseling sessions. At the first session, twenty-severn self-report and three clinical observations are rated on a four-point frequency scale (from “not at all” to “all the time”). At the second session, seventeen additional items are rated on the same scale. Items are a combination of emotional problems (e.g., “Do you feel filled with bad feelings that want to burst out?”); maladaptive behaviors (“Do you take more alcohol than you did before?”); an item about health problems (“Do you experience health problems?”; followed by a checklist of specific health problems following it); and an item about functional impairment (“Do you feel you are being as productive as you can be?”). This scale covers many of the symptoms, behaviors, and functional problems common to individuals who have experienced significant stressors (including potentially traumatic events), including a specific suicidal ideation item. The scale is not specific to survivors of political violence, only trauma. The notable strength of that measure is the inclusion of clinical observations. Structured multi￾source assessment is always an improvement over structured single-source assessment. That mixed methods approach to assessment should be a piece of any thorough screening. Two drawbacks are worth noting. First, there does not seem to be any clinical division between the types of items asked about in the first session and second session. Administering a measure over two sessions introduces some threat to the reliability of findings, as mood may have changed substantially from first session and second; a summary score including responses from all items may therefore reflect two different mood states, and there is no way to check this, given that questions are different between session one and two. The other criticism of the measure is the lack of an explicit time period in which to anchor responses. Again, as with the CAPS Peru’s scale, it is possible that the instructions to respondents regarding the timeframe for their responses were related by the staff administering the screenings. If so, it should be included in the measure itself. Sach! IRPEC Measure That measure is a twenty-four-item scale that uses a four-point severity scale (from “not at all” to “extremely”) and is divided into seven “Anxiety Symptoms,” ten “Depression Symptoms,” and seven “Additional” items. The content of the first two of those sections seems taken (almost word-for-word) from items from the Hopkins Symptom Checklist. The third section includes a 43 suicidal ideation item (“Suicidal thinking”); hyperarousal symptoms (e.g., “Feeling irritable”); and one symptom that is culturally specific to the region (“Sinking of heart”). That measure is specific to anxiety and depression, with some symptoms that characterize PTSD (i.e., that distinguish PTSD from depression and anxiety). Unlike the other measures derived from IRPEC, the measure does not include assessments of functioning or maladaptive behaviors. This may not be a problem for Sach! given its needs, but is inconsistent with the stated values of IRPEC. In addition, and like the measures created by CAPS and IMLU, the measure provides no timeframe in which respondents are supposed to anchor their responses. This should be made explicit on the measure itself. TPO Cambodia’s Self-Help Blue Cloth Measure “Blue Cloth” is a fourteen-item measure with a five-point frequency scale (from “None of the time” to “All of the time”) that instructs respondents to use the timeframe “Last month” and is designed to be administered twice, before and after group interventions. Items include emotional states (e.g., “fear); somatic items (“headache/dizziness”); maladaptive behaviors (“drinking alcohol”), and functioning (“difficulty conducting daily activities”). A version of the Blue Cloth assessment existed before IRPEC began working with TPO. IRPEC assisted TPO staff in checking the validity of the items, and ultimately the measure was reduced from twenty items to fourteen. Notable is the context of administration of Blue Cloth: respondents give their responses in a group context, making the measure a self-report measure with the potential of observer report. Soethara Chimm, director of TPO, explained that for those items that are socially maladaptive (e.g., “gambling”), group members are quick to correct respondents when they underreport. Strengths of Blue Cloth include the pre/post design built into the measure and the range of symptoms, behaviors, and functional indicators included in a relatively brief measure. The mode of group administration may also be seen as a strength, although it is unclear how multi-observer reporting translates to those items which inquire about internal states (e.g., “hopelessness about the future”). The one drawback of Blue Cloth is the use of five items that include more than one indicator. Those items include forward slashes between two conceptually similar but separate indicators (e.g., “headache/dizziness” and “worry/suffering/too much thinking”). It may be that the subsequent indicators are meant to specify the first one listed; if that is the case, it should be made explicit. As presented, it is unclear how a respondent who feels that he or she thinks too much but might not feel that he or she is suffering should respond to “worry/suffering/too much thinking.” However, it may be the case that those five items are accurate translations of single phenomena in Khmer, in which case this criticism is not relevant. Measures Summary In general, measures developed through IRPEC follow standard guidelines of measure design, with the one exception of unclear response timeframe in three of the four measures. Notably, there is a breadth of types of indicators inquired about in three of the four measures (a strength), and it is remarkable how similar many of those indicators are across programs. 44 More information on how each measure is scored would be needed to judge the specific application of those measures (e.g., are the different types of indicators examined as one construct, or as separate phenomena), but even if all were scored as a summary score (i.e., as a single indicator of well-being), this would still be useful for evaluation. 8.5 Building an Effective Regional Infrastructure for Intervention Research Goals (c), (f), and (g) concern the development of regional centers for intervention research and collaboration efforts between those programs and other researchers in the region. In the 2004 explication of goals, IRPEC’s goals are described as “ambitious” (page 1); goals (c), (f), and (g) are the most ambitious of these, particularly as IRPEC is to be completed in only six years. Many torture treatment organizations in the United States have spent years attempting to link evaluation with treatment, with limited success. To teach low-resource organizations in less￾developed countries how to effectively incorporate evaluation into clinical practice, and then to turn them into regional resources for other organizations in a shorter period of time is ambitious indeed. Intervention research reported includes the mixed-methods study currently in the data analysis stage at CAPS Peru using the newly developed measure at intake and follow-up periods (i.e., pre- and post-testing) and comparing results to qualitative investigation with patients and therapists’ perspectives on therapeutic recovery (discussed above), a qualitative study at TPO of baksbat (“soul loss”), a culturally-bounded Cambodian idiom of post-traumatic distress, followed by the development of an appropriate intervention and pre- and post-testing, and continued pre- and post-testing of those involved in community mental health groups; and the beginning of the use of “rolling control groups” at CVT/CAPS Sierra Leone, which will be used to judge program effectiveness with temporarily non-treated comparison groups. With the exception of the last (which is currently in its initial stages), those projects do not conform to the “randomized control group” gold standards of intervention research design. They may, however, provide important program evaluation data concerning clients’ course of symptoms before, during, and after interventions. (Results of many of those evaluations are available in IRPEC quarterly reports.) Regarding the development of organizations as regional centers, several documents reviewed for this report and reports of IRPEC and TPO staff suggest some success. Deup Channarith of TPO has become something of a local resource for data management and statistical analysis. He has designed and taught a course in data analysis for the Royal University of Phnom Penh Reports, and is currently involved in collaborative research with an American researcher on post-partum depression in Cambodia. In addition, he has been involved in validation research for the Hopkins Symptom Checklist in Cambodia at Po Sang Hospital, and is involved with a similar broader study at four sites around the country. IRPEC quarterly reports include information related to the development of IRPEC sites as important regional resources as well. In 2007, Carlos Saavedra of CAPS Peru submitted program evaluation findings for presentation at the International Congress on Traumatic Stress Conference in Argentina, and those were accepted. Also in 2007 (and continuing into 2008), two staff members from CAPS taught a course in monitoring and evaluation at a local university in 45 Lima. However, during the author’s visit to CAPS Peru, Mr. Saavedra reported that although he and other staff at CAPS Peru had been excited to share experiences and techniques with other regional centers at the beginning of the project, they were disappointed that this element of the work was dropped after the first year of the project. He added that he believed that centers in Guatemala, Colombia, and Argentina would be good collaborators for research projects. Dr. Jibaja reported that CAPS is very active within the Latin American network of torture treatment centers, suggesting that CAPS Peru is well-placed to disseminate research findings and techniques. Mr. Saavedra also reported that he was involved in an IRPEC-sponsored exchange with IMLU in Kenya, in which he worked to help IMLU staff members develop their own scale. He reported that although he enjoyed the exchange of ideas, there were some problems with communication, and that the psychologist supported by IRPEC was compromised in terms of the research she could commit to by the multiple roles she was meant to fulfill. Quarterly reports indicate that CVT/CAPS Sierra Leone regularly share personnel for monitoring and evaluation with CVT International Programs in Liberia. Although the structure of second￾tier programs was removed from the IRPEC model in 2006, some participating organizations’ activities are indicative of the degree to which they have become regional centers. For evaluation purposes, it would be helpful to have such activities made explicit in proposals (as they were in the initial 2004 proposal). 8.6 Collateral Benefits The infusion of funds and training into organizations often results in benefits that were not planned, and those associated with IRPEC are no exception. At CAPS Peru those included the integration of monitoring and evaluation into all aspects of activities adopted following its participation in IRPEC. Carlos Saavedra reported that because of the funds made available to hire him and those that helped pay for Erica Cuba and Haydee Antón, data collection became an integral part of the community activities begun by CAPS approximately the same time IRPEC began. For example, in order to better monitor children’s therapeutic playgroups (part of CAPS’ work in provincial communities throughout Peru), Mr. Saavedra helped CAPS staff design an observation form to evaluate change in children’s behavior over the course of those groups. In addition, Mr. Saavedra has been important in supervising monitoring and evaluation staff in the use of CAPS Peru’s database. That database was developed with funds from USAID via the Victims of Torture Fund, and expanded upon through the years. It contains information on clients in therapy, community-based activities, and educational activities undertaken by CAPS staff. CAPS uses information from that database to inform therapists of their progress, funders of the nature of CAPS clients and the impact of their investments, and the general public of the types of problems encountered in the torture treatment field. The use of that impressive tool would not have been possible were it not for the presence of a well-trained researcher like Mr. Saavedra. For TPO, Deup Channarith has raised the profile of TPO in much the same way as has the presence of THI. As noted above, Mr. Channarith has been involved with a number of research projects outside of TPO and has taught at the Royal University of Phnom Penh. The collateral benefit is thus a reinforced reputation. 46 One area that might have been better formalized concerns connecting programs to funding for research and evaluation. Carlos Jibaja of CAPS Peru noted that one of the most helpful collateral benefits was a reference letter provided them by IRPEC to the United States Institute of Peace (USIP). As a result, USIP is currently funding researchers at CAPS Peru to complete a study of the children of incarcerated mothers in Peru. Dr. Jibaja felt that putting more emphasis in IRPEC on connecting to funders would have been extremely helpful. 8.7 Conclusions, IRPEC IRPEC should be seen as an impressive, if qualified, success. Rather than simply giving participating organizations the tools for analysis, IRPEC has built capacity in program evaluation and research by engaging programs in the creation of those tools. In this way they have guided organizations in the developing world to ask their own questions, find their own answers, and take their place in a field dominated by well-funded North American and European groups. To a certain degree, IRPEC has provided a base education that might be available in courses on assessment and evaluation in university settings, but it has done so in the real-world setting of psychosocial programming. By no means have IRPEC staff reached their goals to the extent that they would have liked to. But, they have successfully applied the values that were set out in 2004, and have provided participating organizations with important data that will serve them well in applying for funding from many sources in North America and Europe. Additionally, they have succeeded in establishing a base for doing intervention research in three participating organizations (CAPS Peru, TPO, and CVT/CAPS Sierra Leone). They have accomplished many of the goals that were set out in the 2004 and 2006 proposals in work with those three organizations, and to a lesser extent with the two others (IMLU and Sach!) they have worked with. 8.8 Recommendations, IRPEC 8.8.1 The organizational requirements to collaborate in future research and program evaluation capacity-building efforts should include: 1. a record of at least three years of mental health services 2. at least one staff member whose job description includes (at least in part) collecting, managing, or monitoring data within the organization 3. established interest in research and evaluation on the part of the executive director 4. a commitment to collaborating with educational and research professionals outside of their organizations 5. a stated commitment using empirical data to review and develop clinical services 8.8.2 IRPEC has shown that several practices work well. These should be continued. They include: 47 1. a comprehensive assessment of participating organizations’ program goals for program evaluation and research 2. the inclusion of considering local idioms of distress in designing measures 3. ethnographic methods used to design measures 4. supporting local specialists in data management and analysis through salary support and regular training 8.8.3 Future efforts in monitoring data analysis should include regular demonstrations to program staff of skills relevant to the particular needs of the program 8.8.4 Future efforts in measure development should include a review of standard elements of measurement design (e.g., including a timeframe) as part of advising elements of the design process 8.8.5. Future efforts in establishing research infrastructure should include concrete indicators of regional centers’ success in proposals (collaborations with other centers and university courses given by staff from participating organizations, etc.) 8.8.6 Future efforts should include an emphasis on connecting participating centers to foundations and other funding sources that fund program evaluation and research. 48 9. Relevant Issues Common to Targeted Capacity-Building Programs Several issues common to THI and IRPEC deserve special discussion. 9.1 Situating THI and IRPEC within the ICB Framework Although situated within the larger ICB program, there are key differences between those targeted programs and general ICB subgrant-funded projects. The primary difference is scale. This factor has direct consequences for expected impacts (i.e., targeted programs are expected to impact participating organizations much more than subgrant projects); monitoring and evaluation of those programs (i.e., they are monitored much more frequently and with greater intensity); and support. Those programs do, however, build upon the ICB administrative base in certain ways. For example, IRPEC relies upon a certain base level of computer technology facilitated by ICB technology grants. The reporting standards taught by ICB are required of organizations involved in THI and IRPEC. If either THI or IRPEC were to be separated from the ICB framework, it is clear that organizations would have to have such organizational resources already in place, or that an administrative capacity-building component would have to be undertaken before beginning the THI or IRPEC work in earnest. 9.2 Human Rights versus Healthcare Focus at Implementing Organizations Is it better to orient targeted capacity building programs toward organizations that focus primarily on human rights, or toward organizations that have focus primarily on health care? IRPEC and THI are models for clinically oriented programs. IRPEC requires fairly established mental health programs in order to succeed. The application of those models across programs with different foci is perhaps best exemplified by the comparison of IRPEC’s experience with IMLU to the experiences with TPO and CAPS Peru and CVT/CAPS Sierra Leone. For IMLU, mental health counseling was an additional service to their human rights work, and thus efforts to evaluate those services were always seen as a secondary priority. The other three programs considered their primary mission to be the provision of mental health services, and thus evaluating these services was their first priority. This is not to say that there may never be a program with human rights as its primary focus that might not find IRPEC useful (and indeed, it appears that IMLU has retained some of the record-keeping begun with IRPEC), only that the history of IRPEC would suggest that the likelihood of healthcare organizations successfully building research and evaluation capacity is higher. It might be tempting to view THI’s model of clinical education as a perfect opportunity for a human rights organization to build clinical services. However, as mentioned above (section 7.7) central to the THI model is the notion that an implementing organization can deliver some degree of supervision and general guidance to core group members. Without experience in mental health, it would be difficult to see how those provisions could be met. Of course, without a comparison it is difficult to draw any conclusions. Still, it seems reasonable to assume that as 49 part of the center of a mental health training model (see the innermost concentric circle in Figure 7.1) it would be important to have a fair amount of experience in mental health. That is more likely to be the case with organizations that make healthcare their focus. In conversations with clinical staff at CAPS Peru, one of the organization’s founders, Pilar Raffo, commented that the marriage of mental health and human rights suffers from a difference over the role of boundaries. Human rights workers often aid others without regard to the professional boundaries between helper and client, while in psychology these boundaries are fundamental to therapeutic practice. Although not mutually exclusive, work that includes human rights and psychotherapy therefore involves a basic tension. This tension is only resolved by an organization choosing one perspective or the other. 9.3 Accommodating Language Surprisingly, the challenges of teaching across languages were largely absent from documents reviewed, and even in discussion of THI and IRPEC. During the training observed at TPO Cambodia, language was an obvious barrier, overcome only to some degree by hiring an interpreter. The interpreted lectures observed had a stilted quality to them, despite the lecturers’ best efforts. Carlos Saavedra of CAPS Peru (who speaks English well) noted that variable fluency in English between centers was an impediment during the group telephonic supervision in IRPEC, suggesting that language barriers were likely a challenge for instruction in measurement and design as well. Good interpretation only partially solves the problem. Because psychology is largely a Euro￾American phenomenon, speakers of many languages borrow words from European languages when they discuss the mental health problems. For instance, the word “trauma” was mentioned several times in the Khmer interpretation of the THI training. It was unclear whether there had been any attempt to find a suitable Khmer word, or whether the English word had been explained at some point and used subsequently. Of course, for some organizations in countries where European languages are used (e.g., CAPS Peru), this issue may be less relevant. As both THI and IRPEC at least include pedagogical components, identifying the specific relevant psychological and evaluation terms in the target language would seem to be a critical component of any assessment effort. Learning activities should minimize the use of lecturing where feasible. 9.4 Dependency Issue for Implementing Organization in the Context of Multiple ICB Programs The Center for Victims of Torture has stated that ICB programs should not foster dependency. However, for centers that are involved in both general capacity-building support through ICB and targeted programs IRPEC and THI, the issue of whether the administering organization is or is not fostering dependency is complicated. Dependency is likely a continuous phenomenon (i.e., there are degrees of dependency), and the level of dependency may be based on the interaction between timing of specific programs and length of funding periods. For example, according to 50 “2006-2010 USAID Cost Extension Assistance Grants,” CAPS (Peru) has been allocated $192,411 since 2001, or an average of $24,051 a year; this has included five years of ICB (no allocation of money is listed following 2006) and then three years of IRPEC. TPO Cambodia, on the other hand, has been allocated $266,801 for the period beginning in 2005, an average of $66,700 a year; this has included money for ICB, IRPEC, and THI simultaneously.10 CAPS Peru’s funding history—receiving ICB subgrants and then receiving IRPEC support—is preferable (in terms of the goal of non-dependency) to the simultaneous delivery of money via three programs to TPO. Chief Administrator Jacqueli Fontela explained that only one full-time position (researcher Carols Saavedera) was funded with IRPEC funds and none by ICB, and that the end of IRPEC funding would only result in the loss of that position alone (other portions of salaries for monitoring and evaluation personnel were minimal and would be augmented with other funding sources). In contrast, TPO has used IRPEC and ICB funds to pay for at least three full-time positions. With the sizeable “investment” of money over a relatively short period of time, it seems unlikely that TPO will structure the use of that money consistent with the short￾term scope of the funding. As director of TPO, Dr. Sotheara reported when asked about future plans, “This is just the beginning of our relationship with CVT.” In fact, the relationship is supposed to end in 2010. The difference in funding per year is sizeable in terms of planning and expectations of specific programs for funding, even more so when taking into account differences in standards of living between Peru and Cambodia (development indices place Peru well above Cambodia). 9.5 Reporting Practices The main challenge in undertaking this evaluation has been reviewing the vast amount of documentation completed over the grant period. THI and IRPEC staff have been meticulous in reporting details in the numerous proposals, quarterly reports, and memoranda that have gone into initiating and carrying out such intensive programs. But, they have not provided summary documentation to bring all of these documents together. In a sense, staff have collected a large amount of raw data on their activities over the reporting period, but they have not provided the descriptive summary data that helps outsiders make sense of those activities. A visual timeline (perhaps updated annually) or some related form of cumulative reporting tool would have provided a global historical perspective that could then be used to track changes over time in proposal, activities, and progress. Interestingly, THI and IRPEC staff are able to provide this perspective in conversation. To staff, quarterly reports may seem to provide a complete picture precisely because they have this perspective; however, to those reviewing their work, quarterly reports and memoranda can be difficult to follow over time. 10 Even if THI ($140,000) had not been hosted by TPO, the daily average would still be substantially higher than Peru—an average of $31,700 a year—because the ICB and IRPEC money would still be delivered simultaneously. 51 9.6 Recommendations 9.6.1 Organizations that are candidates for targeted capacity-building programs must include the provision of mental health services as a key piece of their mission. The provision of mental health services may be the organization’s primary mission, or it may be placed on an equal footing with human rights; it should not be a secondary mission. 9.6.2 How the multiple challenges of how language will be handled should be made explicit, and translation fidelity should be a central goal and evaluation point for future projects. Learning activities should minimize the use of lecturing where feasible. 9.6.3 The following recommendations should be followed if a goal of the administering agency is to minimize dependency on program funding: 1. The administering organization should set yearly funding ceilings for subgrantees involved in targeted capacity programs and describe specific plans for how those subgrantees will transition out of those structures. 2. The administering organization should provide substantial justification for granting one partner general capacity-building funding and targeted program funding simultaneously. This justification should include details about how funding multiple programs will not foster the program’s dependency on the administering organization. 9.6.4 Some form of cumulative reporting tool should be adopted for multi-year, multi-component programs. This tool might take the form of a timeline, flow chart, or other graphic device. 52 ANNEX 1: SCOPE OF WORK Final-term Evaluation of International Capacity Building (ICB) Project SOW for Jennifer Whatley: BACKGROUND The Center for Victims of Torture in Minneapolis, Minnesota has implemented a cooperative agreement with USAID since October 2000. This cooperative agreement was recently amended to extend the end date to June 30, 2010 and is entitled “Capacity Building of Foreign Treatment Centers” (referred to internally at CVT as the International Capacity Building or ICB Project). The goal of the ICB project is to expand available global resources, professional and service capacities, in order to heal and support survivors of torture, including encouraging new approaches for the prevention of torture and restoration of civil society made possible by the strategic position of torture treatment centers. These centers are at various stages of development and institutional needs. CVT has undertaken a program to evolve the capacity of these centers through assessments, operational support grants, technology grants, organizational development consultants, training workshops, and on-site coaching. The program is managed through the headquarters office of CVT in St. Paul, Minnesota. The project management team is responsible for administering the grant, providing and managing technical support and tracking financial transactions. CURRENT OBJECTIVES (July 1, 2006 – June 30, 2010) 1. “ICB Center Strengthening”: Strengthen the capacity of global torture treatment centers, both in terms of organizational viability as well as the delivery of services and advocacy for human rights (Objectives 2,3 and 4 covered under Mental Health Expert, Andrew Rasmussen) 2. “Trauma Healing Initiative (THI)-Cambodia”: Through a local torture treatment center, TPO Cambodia, build clinical and public education capacity among a network of community torture/trauma treatment resources in Cambodia to more broadly impact torture survivors; 3. “International Research and Program Evaluation Collaboration (IRPEC)”: Continue to strengthen research capabilities of project partner centers to enable them to measure the impact of rehabilitation interventions on the functioning and well-being of torture survivors; 4. “CVT - Sierra Leone”: Continue to provide extensive clinical training and supervision to a group of experienced Psychosocial Counselors (PSCs) who in turn offer direct counseling services to survivors of torture in Kono District of Sierra Leone, while fostering the development of “CAPS,” the Community Association for Psychosocial Services, an independent torture rehabilitation/resource center in the Kono District. (2006 - 2008 only) 53 Previously, in 2002, the ICB project conducted its own Internal Retrospective Review, which was then shared with two external evaluators who visited CVT’s headquarters to conduct a management review. In 2004 another Internal Retrospective Review was conducted by ICB Project staff. Since then, the project has continued and expanded to include 3 additional objectives (see above). PURPOSE The purpose of this consultancy is to conduct a mid-term evaluation, as required by USAID under the agreement, to assess performance of CVT in completing the terms of its agreement. This entails: 1. Examine compliance with the terms of the agreement. 2. Determine results achieved. 3. Identify areas for improvement. 4. Provide insight to the following broader questions about the ICB project: a. What is the right size of an organization for the ICB project to be working with? (If the organization is too small, change takes a long time and requires a large amount of resources). What is a good balance between number of partners and amount of resources for each partner? b. “Graduation” – how long should the ICB center work with a partner? What is the evidence of what will remain after the USAID project is over? Expanded areas? What skills/practices do they continue? New practices/components kept? c. Human rights / health care split: Is it better to develop the clinical side of a human rights/torture treatment organization? Or add torture treatment/human rights to a general mental health care organization (eg as we’ve done with TPO in Cambodia)? Do humanitarian organizations have a role? How do torture treatment services relate to rule of law activities? d. Should the ICB project keep a global focus or instead focus on one region? e. Should the ICB project (Objective 1) keep a holistic focus (organizational development, clinical programs, advocacy and technology) or continue more focused capacity building (Objective 2: THI Cambodia focused on building counseling skills and Objective 3: IRPEC focuses on building program evaluation/research skills). What techniques are trainable and sustainable? f. Could the ICB project work more strategically with CVT’s offices overseas (funded through PRM) to work with the staff to develop their own organization? See CAPS in Sierra Leone (current ICB partner) and LAPS in Liberia (not an ICB partner). ACTIVITIES The evaluation is divided into two major tasks or phases. They are: 1. A review of headquarters. The consultants will review and follow up on the internal retrospective review (1 above), review key internal documents, and review project interventions and 54 assessment measures. As the previous external evaluation in 2002 already focused on compliance and administration, this headquarter review will provide background research for the consultants, and be focused on examining programmatic tools and effectiveness. a. Program accomplishments under the grant. Extensive discussion and review of project accomplishments will be available through CVT’s internal retrospective review document. The consultants should include follow up to the internal retrospective review document in order to comment on the outputs, outcomes, and process of capacity building. b. Tools of capacity building. The consultants should comment on the tools utilized to build capacity and measure capacity building (e.g., organizational assessment instruments, training). c. Project management. The consultants should comment on organizational structure, management responsibilities. d. Field support. The management team should determine the quality of field support from headquarters. 2. Field visits to 2 – 3 selected treatment center sites. Two or three countries will be identified to review the status of capacity of foreign treatment centers resulting from interventions of CVT. The field visits should assess the change in capacity as a result of CVT’s support under the project, examine project management from the field perspective and financial controls. How do partners feel the ICB project facilitated their development? Recommended sites to be discussed include: Cambodia, Peru, Sierra Leone SOW for Andrew Rasmussen: PURPOSE The purpose of this consultancy is to conduct a late-term evaluation, as required by USAID under the agreement, to examine the 4 capacity-building strategies and lessons learned, and provide insight to the following broader questions about each of the 4 strategies: 1. “ICB Center Strengthening” a) Human rights / health care split: Is it better to develop the clinical side of a human rights/torture treatment organization? Or add torture treatment/human rights to a general mental health care organization (eg as we’ve done with TPO in Cambodia)? This would be a critical aspect to consider. It would be helpful to have CVT construct a matrix, that might help us to better understand the real overall nature of the partners and program. 2. “Trauma Healing Initiative (THI)-Cambodia”: What is the target population of this program? Just the folks who are preparing to testify at the tribunals or others? Does CVT know what the patient population mix is? Do the Tribunal patients require counseling and assistance now, or is there a timeline when people are anticipated to come under the stress of re-trauma? a) What role does manualization play in sustaining and disseminating the clinical training? (Is it worth the effort to have an interactive/A-V version?) 55 b) THI provides a broader academic and skills training vs. training to a specific trauma intervention. What are the implications of this in terms of capacity-building? From a capacity-building standpoint can we get to the same point of individual judgment? c) How does the implementing agency (TPO) view the value of THI to its aims? 3. “International Research and Program Evaluation Collaboration (IRPEC)”: a) What needs to be in place at an organization (how developed should an organization be) for this objective to be the most effective? This is important and a corollary to the "size isn't everything" question raised above. Can IRPEC work in an organization that was not a part of the more holistic capacity building approach of ‘ICB Center Strengthening’? b) Would it be better if there was a full-time on-site expert, instead of CVT’s month-long visits twice a year? ACTIVITIES The late-term evaluation is divided into two major tasks or phases. They are: 1. A visit to CVT headquarters in Minneapolis. The consultants will review and follow up on the internal retrospective review (1 above) and review project interventions and assessment measures. As the previous external evaluation in 2002 already focused on compliance and administration, this headquarter review will provide background research for the consultants, and be focused on examining programmatic strategies, tools, accomplishments and effectiveness with CVT staff. 2. Field visits to at least one selected treatment center site and possible a second to be named later. Two or three countries will be identified to review the status of the center resulting from interventions of CVT. The field visits should assess the change in capacity as a result of CVT’s support under the project. How do partners feel the ICB project facilitated their development? Recommended sites include Cambodia and Peru. DELIVERABLES A written report will be submitted to USAID. The report should include: Description of the models and what has happened using that model Challenges Results Which of these strategies hold the best promise? What are the best aspects of each strategy? 56 ANNEX 2: MEETINGS AND INTERVIEWS U.S Agency for International Development (DCHA/DG) Lloyd Feinberg Cathy Savino Center for Victims of Torture, Minneapolis, MN Ally Beckman, Clinical Social Worker Pete Dross, Director of Policy and Development Jon Hubbard, Director of Research Edie Lewison, International Services Coordinator My Lo, ICB Project Coordinator Linda Nielsen, International Clinical Consultant Neal Porter, Director of International Services Pamela Kriege Santoso, ICB Project Manager Carol White, Trauma Healing Initiative Cambodia Coordinator Beth Wickum, Director of Volunteer Services African Center for the Treatment and Rehabilitation of Torture Victims, Kampala, Uganda Ruth Bonabaana, Finance & Administrative Manager Faith Bothuwok, Trauma Counselor Stephen Kadaali, former CEO Fred Muzira, Social Worker & Acting Program Manager Samuel Nsubuga, CEO Independent Medico-Legal Unit, Nairobi, Kenya Wanjiru Gathuru, Finance and Administration Officer Dinah Kituyi, Counselling Psychologist Sam Mohichi, Executive Director (outgoing) George Ngatia, IT & Research Officer Rose Oray, Administrative Assistant 57 ANNEX 3: KEY DOCUMENTS REVIEWED African Centre for Treatment and Rehabilitation of Torture Victims. Annual Report 2005. African Centre for Treatment and Rehabilitation of Torture Victims. Annual Report 2006. African Centre for Treatment and Rehabilitation of Torture Victims. Annual Report 2007. African Centre for Treatment and Rehabilitation of Torture Victims. Annual Report 2008. Center for Victims of Torture. “Capacity Building for Foreign Treatment Centers – International Capacity Building Project, Cost Extension Request.” Submitted to USAID February 2004. Center for Victims of Torture. “Capacity Building for Foreign Treatment Centers – International Capacity Building Project, Cost Extension Request.” Submitted to USAID May 2006, pp. 35-41. Center for Victims of Torture. Excerpts from funding request submitted to USAID in 2000. Center for Victims of Torture. International Capacity Building Project: Project Partner Manual July 2008-June 2010. Center for Victims of Torture. “Organizational Matrix Report by Country.” August 2004 Center for Victims of Torture. “Organizational Assessment Matrix Compiled All Centers.” August 2009 Center for Victims of Torture. “USAID Detailed Implementation Plan: July 1, 2006-June 30, 2008.” Ellmaker, Charles and Herve Rakoto Razafinbahini, MD. “African Centre for the Treatment and Rehabilitation of Torture Victims: Field Assessment – April 17-25, 2001 – Kampala, Uganda. Center for Victims of Torture. Geoffe & Associates. Independent Medico-Legal Unit (IMLU) Report and Financial Statements for the Period Ended 31 December 2007. September 2008 58 Geoffe & Associates. Independent Medico-Legal Unit (IMLU) Report and Financial Statements for the Period Ended 31 December 2008. April 2009 Christine Hjelt and Jack Hjelt. Management Review of the ICB Project. 2002. Various ICB Supported Centers. Emailed responses to ICB August 2007 Internal Evaluation Survey. Responses from the centers in Cambodia (TPO), Ethiopia, India, Kosovo, Namibia, Palestine, Peru, Romania, Rwanda, South Africa, Uganda were reviewed. Nemtanu, Raluca. Participatory Institutional Analysis Instrument: A Comparative Analysis of Similar Instruments, Lessons Learnt from the Field, Recommendations for Future Use, World Learning, September 2006. World Learning. Institutional Analysis Instrument: A Participatory NGO Development Tool. 2000. 59 ANNEX 4: TORTURE TREATMENT CENTERS SUPPORTED BY ICB Center for Rehabilitation of Torture Survivors (CRTS), BANGLADESH Assistance Center for Torture Survivors (ACET), BULGARIA Cambodian League for the Promotion and Defense of Human Rights (LICADHO), CAMBODIA Transcultural Psychosocial Organization (TPO - Cambodia), CAMBODIA Center for Rehabilitation and Abolition of Trauma (CRAT), CAMEROON Rehabilitation Center for Victims of Torture in Ethiopia (RCVTE), ETHIOPIA Equipo de Estudios Comunitarios y Accion Psicosocial (ECAP), GUATEMALA Shubhodoaya Center for Rehabilitation of Victims of Torture and Violence (SCRVTV), part of SOSRAC, INDIA Independent Medico-Legal Unit (IMLU), KENYA Kosova Rehabilitation Center for Torture Victims (KRCT), KOSOVO People's Education, Assistance, and Counselling for Empowerment (PEACE), NAMIBIA Center for Victims of Torture (CVICT), NEPAL Struggle for Change (SACH), PAKISTAN Treatment and Rehabilitation Center for Victims of Torture (TRC), PALESTINE Centro de Atencion Psicosocial (CAPS), PERU ICAR Foundation, Medical Rehabilitation Center for Torture Victims, ROMANIA 60 Forum des Activistes Contre la Torture (FACT), RWANDA Community Association for Psychosocial Services (CAPS), SIERRA LEONE Trauma Center for Survivors of Violence and Torture (TCSVT), SOUTH AFRICA African Center for the Treatment and Rehabilitation of Torture Victims (ACTV), UGANDA U.S. Agency for International Development Victims of Torture Fund 1300 Pennsylvania Avenue, NW, Suite 700 Washington, DC 20523 Tel: (202) 789-1500 Fax: (202) 789-1601 www.usaid.gov