MANAGEMENT SYSTEMS INTERNATIONAL 600 Water Street, S.W. 202/484-7170 Washington, D.C. 20024 Fax: 202/488-0754 USA mfanning@msi-inc.com EVALUATION OF STRATEGIC OBJECTIVE 3.1 USAID HUMANITARIAN ASSISTANCE IN CONFLICT-AFFECTED AREAS IN AZERBAIJAN FINAL REPORT December 7, 2002 MSI/MetaMetrics Inc. Technical Evaluation Team Leo T. Surla, Jr., Team Leader Christopher Bladen, Health Analyst David A. Smith, Management Analyst FINAL REPORT i USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation TABLE OF CONTENTS ACKNOWLEDGEMENTS ....................................................................................................................iv LIST OF ACRONYMS ...........................................................................................................................v KEY DEFINITIONS AND IDP MAP.....................................................................................................vi EXECUTIVE SUMMARY.....................................................................................................................ix SECTION 1 INTRODUCTION.............................................................................................................1 1.1 STRATEGIC OBJECTIVE BACKGROUND.......................................................................1 1.2 EVALUATION PURPOSE..................................................................................................2 1.3 COUNTRY CONTEXT........................................................................................................2 1.3.1 Armenian-Azerbaijani Conflict and the Peace Process.........................................3 1.3.2 Governmental Structure ......................................................................................4 1.3.3 Economy ............................................................................................................5 1.4 EVALUATION APPROACH AND ACTIVITIES ...............................................................7 SECTION 2 HEALTH PROGRAMS......................................................................................................9 2.1 DEVELOPMENTS SINCE INDEPENDENCE ....................................................................9 2.2 PROGRAM/PROJECT DESCRIPTIONS...........................................................................10 2.2.1 American International Health Alliance (AIHA) ...............................................10 2.2.2 International Medical Corps (IMC)...................................................................11 2.2.3 Adventist Development and Relief Agency (ADRA).........................................12 2.2.4 Pathfinder International.....................................................................................12 2.2.5 Community Habitat Foundation, CHF...............................................................13 2.2.6 International Rescue Committee (IRC)..............................................................13 2.2.7 Save the Children..............................................................................................14 2.3 PROGRAM EFFECTIVENESS .........................................................................................15 2.4 ISSUES/CONSTRAINTS...................................................................................................17 2.5 SUSTAINABILITY/INTEGRATION/TRANSITION ........................................................18 2.6 POLICY AND RELATED ISSUES....................................................................................19 SECTION 3 ECONOMIC OPPORTUNITY PROGRAMMING...........................................................21 3.1 SUMMARY OF OVERALL PARTNER PROGRAMS......................................................21 3.2 ECONOMIC PROGRAM REVIEW...................................................................................21 3.3 FINDINGS AND CONCLUSIONS ....................................................................................23 3.4 RECOMMENDATIONS ....................................................................................................23 SECTION 4 COMMUNITY DEVELOPMENT....................................................................................24 4.1 COMMUNITY MOBILIZATION AND DEVELOPMENT................................................24 4.2 FINDINGS AND CONCLUSIONS ....................................................................................24 4.3 RECOMMENDATIONS....................................................................................................25 FINAL REPORT ii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 5 NAXCIVAN AUTONOMOUS REPUBLIC.....................................................................27 5.1 PROGRAM/PROJECT DESCRIPTION.............................................................................28 5.2 PROGRAM EFFECTIVENESS .........................................................................................30 5.3 ISSUES/CONSTRAINTS...................................................................................................31 5.4 SUSTAINABILITY/INTEGRATION/TRANSITION ........................................................31 5.5 POLICY AND RELATED ISSUES....................................................................................33 SECTION 6 REVIEW OF AHAP MECHANISM.................................................................................34 6.1 PROGRAM OVERVIEW...................................................................................................34 6.1.1 Cooperative Agreement ....................................................................................34 6.1.2 Subgrants..........................................................................................................34 6.2 PROGRAM EFFECTIVENESS .........................................................................................35 6.2.1 Achievement of USAID Objectives...................................................................36 6.2.2 Mobilizing Activities ........................................................................................37 6.2.3 Selection...........................................................................................................37 6.2.4 Monitoring........................................................................................................38 6.2.5 Program Reporting............................................................................................39 6.2.6 Quality of Data .................................................................................................39 6.2.7 Financial Reporting...........................................................................................40 6.2.8 Compliance.......................................................................................................40 6.2.9 Use of Data for Decision-making ......................................................................41 6.2.10 Coordination.....................................................................................................42 6.2.11 Trust.................................................................................................................44 6.2.12 Grantee Attitudes..............................................................................................45 6.3 PROGRAM COST .............................................................................................................45 6.3.1 Staffing.............................................................................................................45 6.3.2 Cost Implication of Umbrella vs. Direct Management .......................................46 6.3.3 Value-added .....................................................................................................47 6.4 SUMMARY.......................................................................................................................49 6.5 KEY FINDINGS ................................................................................................................49 6.6 RECOMMENDATIONS....................................................................................................51 SECTION 7 FINDINGS AND RECOMMENDATIONS ......................................................................53 7.1 EVALUATION QUESTIONS............................................................................................53 7.1.1 Critical Humanitarian Needs.............................................................................53 7.1.2 Transfer of AHAP Economic Opportunity Projects to the Mission S.O. 1.3, Accelerated Growth of Small and Medium-Sized Enterprises in Targeted Areas..............................................................................................54 7.1.3 Program Effectiveness ......................................................................................54 7.1.4 Subgrantee Objectives and Meeting Needs of Targeted Beneficiaries................55 7.1.5 Level of Community/Beneficiary Participation..................................................55 7.1.6 Geographic Appropriateness of S.O. 3.1 Activities............................................56 7.2 HEALTH FINDINGS AND RECOMMENDATIONS........................................................57 7.2.1 Health Program Effectiveness...........................................................................58 7.2.2 Health Issues and Constraints............................................................................59 7.2.3 Health Program Sustainability...........................................................................60 7.2.4 Health Policy and Program Recommendations..................................................61 7.3 ECONOMIC OPPORTUNITY...........................................................................................62 FINAL REPORT iii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 7.3.1 Findings and Conclusions .................................................................................62 7.3.2 Recommendations.............................................................................................63 7.4 COMMUNITY DEVELOPMENT FINDINGS AND RECOMMENDATIONS..................63 7.4.1 Inclusion of Economic Opportunity Projects .....................................................63 7.4.2 Levels of Community Project Funding..............................................................64 7.4.3 Community Mobilization/Participatory Methodologies .....................................64 7.5 AHAP UMBRELLA PROGRAM FINDINGS AND CONCLUSIONS...............................64 7.5.1 Management Structure of the Umbrella Mechanism..........................................65 7.5.2 Umbrella Effectiveness and Innovations............................................................65 7.5.3 Umbrella Issues ................................................................................................65 7.5.4 S.O. 3.1 Strategy Development and Implementation..........................................66 7.5.5 MCI Monitoring and Compliance......................................................................66 7.5.6 Umbrella Mechanism as an Implementing Vehicle............................................67 7.6 UMBRELLA DEVELOPMENT/IMPLEMENTATION RECOMMENDATIONS..............68 7.6.1 Umbrella Review and Program Strategy............................................................68 7.6.2 Staffing and Personnel Resources......................................................................69 7.6.3 Monitoring and Integration ...............................................................................69 7.7 MISSION STRATEGY DEVELOPMENT.........................................................................69 7.7.1 Government Interaction ....................................................................................70 7.7.2 Sustainability....................................................................................................71 7.7.3 Monitoring and Follow up.................................................................................72 ANNEX A DRAFT SELECTED BIBLIOGRAPHY.............................................................................77 ANNEX B INTERVIEW/MEETING/FIELD VISIT SCHEDULE........................................................80 ANNEX C REDUCED HUMAN SUFFERING IN CONFLICT-AFFECTED AREAS IN AZERBAIJAN ...................................................................................................................85 ANNEX D REDUCED HUMAN SUFFERING IN CONFLICT-AFFECTED AREAS IN AZERBAIJAN ...................................................................................................................88 ANNEX E AHAP INFORMATION .....................................................................................................99 FINAL REPORT iv USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ACKNOWLEDGEMENTS The MSI/MetaMetrics Evaluation Team acknowledges that the evaluation effort was strongly enhanced by the commitment and support of the Mission in Azerbaijan. Under the Mission direction of William D. McKinney, arrangements were made by USAID/Azerbaijan staff to coordinate the Baku interviews and facilitate the scheduling of the field visits to the four targeted AHAP regions of Azerbaijan, including the trip to Naxcivan. The Evaluation Team was accompanied on almost all of the field visits by Valerie Ibaan, Humanitarian Response Program Specialist and Gulnara Rahimova, Program Development Specialist, the key program personnel of the Mission with responsibilities for managing the AHAP program. They, additionally, participated in the Team discussions on the field and interview data. Mercy Corps personnel and all eight of the AHAP Partners plus AIHA gave of their valuable time to participate in lengthy interviews and arrange the details of the field trips. In particular, the Evaluation Team singles out Myriam Khoury, Sue Leonard, and Mary Hennigan of Mercy Corps; Beverly Hoover, Slawek Getka, and Azad Budagov of CHF; Adam Sirois and Fuad Hamidzade of IMC, Pamela Husain and Elaine Thomas of IRC; Mark Castellino and Jenny Sequiera of ADRA although nearly the entire AHAP Partners’ field staff played important roles to support the Evaluation Team effort. Our interpreter, Leyla Masma Najafzadeh, was critical to the conduct of individual key informant and focus group interviews in the field and was an important Team player. While this support is acknowledged and extensive cooperation was provided by USAID and AHAP Project personnel, the MSI/MetaMetrics Evaluation Team takes full responsibility for the content of this report. FINAL REPORT v USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation LIST OF ACRONYMS ACDI/VOCA Agriculture Cooperative Development International/Volunteers in Overseas Cooperative Assistance ADRA Adventist Development and Relief Agency AHAP Azerbaijan Humanitarian Assistance Program AIHA American International Health Alliance AIM Agricultural Information Center BDS Business Development Services CBLS Community Based Lending and Saving CDC Centers for Diseases Control and Prevention CEEOP Community Employment and Economic Opportunity Program CERG Community Economic Recovery Group CG Community Group CHF Community Habitat Foundation CHMC Community Health Management Committee EO Economic Opportunity FSA FREEDOM Support Act GOAJ Government of Azerbaijan ICDP Integrated Community Development Program IDP Internally Displaced Person IMC International Medical Corps INGO International Non-Governmental Agency IOM International Organization on Migration IRC International Rescue Committee MCI Mercy Corps International MEDS Monitoring, Evaluation and Design Support MOH Ministry of Health MSI Management Systems International NAR Naxcivan Autonomous Republic NIS Newly Independent States NK Nagorno-Kharabagh OSCE Organization for Security and Cooperation in Europe PMP Performance Monitoring Program PVO Private Voluntary Organization SC Save the Children SFDI Social Fund for the Development of Internally Displaced Persons SII Social Investment Initiative S.O. Strategic Objective UNHCR United Nations High Commission on Refugees USAID United States Agency for International Development VHC Village Health Council WHO World Health Organization FINAL REPORT vi USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation KEY DEFINITIONS AND IDP MAP Some clarifications on terms within the context of AHAP programming and the evolving Mission strategy are presented below. These include the frequently used terms: "transition," "sustainability," and "integration." Community development has at times, for example, been interpreted to mean primarily those organizational activities that resulted in community action groups. “Integration” has political connotations that if not clarified, could be interpreted as integration of IDPs into the local population. Similarly, Azeri translations of “community mobilization” have inferred a military meaning. The following map shows the location of major refugee/ IDP camps in Azerbaijan, and is from http://www.azer.com/aiweb/graphics/maps/majorcities_map.html. Community Development: In the context of this report, community development includes the range of activities beginning with organization and mobilization of interested community members to undertake projects of general community interest. Community development includes the placing of priorities, planning, and implementation of projects. Community Group (CG): Group of individuals selected by community that receive training, group loans, and/or lead in implementation of a community project. Conflict Affected Population: Internally Displaced Persons (IDPs), refugees, and local populations in proximity to and sharing resources with populations of refugees and IDPs. Includes persons in communities of Naxcivan who are affected by the economic blockade and continued restricted access to mainland Azerbaijan. Integration: The Evaluation Team has used alternative wording as appropriate to the evaluation context. Integration, however, is a widely used term within AHAP program implementation. At various times integration can mean coordination and cooperation between Partners, coordination with a Partner’s project components as between loan projects and health projects, and cooperation and information sharing with local authorities such as Municipalities and Ex Com personnel. Internally Displaced Persons (IDPs): Persons who fled the Nagorno-Kharabakh area of Azerbaijan and surrounding seven (7) regions as a result of the Armenian-Azerbaijani Conflict and currently live on Azerbaijan territory. Local Non-Government Municipalities: There are about 5,000 locally elected officials for Chair of the Municipality and Members of the Municipality. These positions, though paid by the national government, are referred to as non-governmental. This terminology may be used to distinguish these persons from the direct representatives of the national government at the local level, the members of the Executive Committee or Ex Com. Local NGOs: Formal and, in Azerbaijan, primarily unregistered groups of interested individuals who provide technical assistance and information on topics of interest in health, community development and economic opportunity, environment, legal issues, and other areas. FINAL REPORT vii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Refugee: Azeri persons who fled from Armenia as a result of the Conflict as well as other conflict affected people who have settled in Azerbaijan from other countries. Sustainability: The general meaning of sustainability is the ability of engaged groups and individuals to continue with project implementation and operations after the end of donor financing and provision of technical assistance. Transition: While the U.S. Government will continue to provide humanitarian assistance to needy Azerbaijan areas, USAID/Azerbaijan also has the program objectives of supporting general economic development and democratic initiatives. The S.O. 3.1 program over the past two years has provided support for health, community development, and economic opportunity. These programs were targeted for the conflict-affected areas of Azerbaijan. “Transition” also has connotations of sustainability as the responsibility for community development projects is assumed by the communities themselves and the Government of Azerbaijan increases its involvement of support to local community development efforts. FINAL REPORT viii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation FINAL REPORT ix USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation EXECUTIVE SUMMARY This evaluation examined USAID/Azerbaijan’s Strategic Objective 3.1, Reduced Human Suffering in Conflict-Affected Areas, evaluation emphasis was on the USAID Azerbaijan Humanitarian Assistance Program (AHAP), which was the major vehicle for providing assistance under this strategic objective. The evaluation reviewed AHAP as a set of activities designed to achieve the strategic objective and as a mechanism for managing those activities. The AHAP was implemented through a cooperative agreement with Mercy Corps International (MCI). It commenced in January 1998 and, as amended, is a six-year activity with a funding level of $45 million that ends in January 2004, an approximate 14 months from the completion of this evaluation. PROGRAM OVERVIEW MCI issued two rounds of requests for grant applications over the past four years in the areas of food, health, shelter, community development, and economic opportunity. Fourteen operational subgrants in the second round were awarded for health, community development, and economic opportunity. Several of these subgrants were extended to be completed by November 2003 and are implemented by eight U.S. based PVO Partners: Agriculture Cooperative Development International/Volunteers in Overseas Cooperative Assistance (ACDI/VOCA), Adventist Development and Relief Agency (ADRA), Community Habitat Foundation (CHF), International Medical Corps (IMC), International Rescue Committee (IRC), Pathfinder International (Pathfinder), Save the Children (SC), and World Vision. These subgrants were for programs in the four USAID conflict-affected areas of Central and South IDP Belt, the Urban area near Baku, and Naxcivan. Strategy Background Although USAID commenced activities in Azerbaijan in 1992, the first formal program strategy started in FY 2000 and was scheduled to end in January 2004. One of the four Strategic Objectives (S.O.) of this current strategy is S.O. 3.1, Reduced Human Suffering in Conflict￾Affected Areas. The USAID/Azerbaijan strategy will be updated to reflect programmatic changes made possible by the January 2002 Presidential waiver of Section 907 of the FREEDOM Support Act (FSA). Although humanitarian assistance was exempted from Section 907 in 1999, AHAP had community development and economic opportunity programs at that time and continued to be constrained by the application of Section 907. Under the waiver of Section 907, subgrant activities under S.O. 3.1 have been able to work openly with the Government of Azerbaijan (GOAJ) and the Mission is now enabled to choose to fund projects and directly be engaged in project coordination and support to GOAJ. Evaluation Purpose The evaluation statement of work provided the following: “The purpose of this evaluation is to review S.O. 3.1, its associated activities, and the performance of AHAP as a management FINAL REPORT x USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation mechanism. The findings and recommendations will inform Mission management and the implementing Partners how effective the current program has been in meeting the strategic objective and where and how possible adjustments could be made to improve program implementation and impact.” Evaluation Activities This final report presents the assessment of the MSI/MetaMetrics Evaluation Team of the humanitarian assistance programming based on the analysis of interviews with MCI, the AHAP subgrant Partners, and information collected during approximately twenty-five field site visits to health, community development, and economic opportunity projects. These site visits included interviews with community groups organized under the activities of the AHAP Partners, with beneficiaries, and with focus groups with no or little contact with the AHAP programming. Field visits were conducted in the Sumgait area near Baku, the IDP zone of Central and Southern Azerbaijan, and in Naxcivan Autonomous Republic. All of these areas have been impacted by the Armenian-Azerbaijani Conflict. On these visits, the Evaluation Team was accompanied by key program personnel of USAID/Azerbaijan. EVALUATION QUESTIONS The evaluation statement of work posed a series of evaluation questions associated with a review of S.O. 3.1, Reduced Human Suffering in Conflict-Affected Areas, to be addressed by the Evaluation Team. Key findings and recommendations in response to these evaluation questions are presented below. Critical Humanitarian Needs Partner subgrantee personnel in the field have indicated that, to their knowledge, the critical humanitarian needs of emergency food and shelter are generally available to all IDPs. There are, however, pockets of greater need, e.g. in Naxcivan and in many communities with deteriorating infrastructure and a lack of economic opportunities. Based on spotty reports of stunting and other diet related problems, in a few areas available food may be of poor nutritional quality. While medical services, at some level, are available, the quality of and access to health care and pharmaceuticals vary widely. Housing conditions range from unheated railroad boxcars to newly constructed individual family houses with electricity and drinking water. In the conflict-affected areas visited by the Evaluation Team, local non-IDP residents as well as the IDPs in these communities needed medical services, electricity, safe drinking water, and improved roads. According to community and IDP leaders that were interviewed, in addition to health and infrastructure needs, school buildings and community centers were high priorities. These were closely followed by agricultural economic opportunity needs (irrigation, salt water drainage, food processing), youth programs (sports fields and gymnasiums), and culturally important ceremonial sites for weddings and funerals. FINAL REPORT xi USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Transfer of AHAP Economic Opportunity Projects to the Mission S.O. 1.3 The Evaluation Team visited several economic opportunity programs that were implemented under S.O. 3.1 in the "IDP belt" of Central and South Azerbaijan. Humanitarian Assistance under S.O. 3.1 can broadly be interpreted as programs that reduce human suffering. As such, the economic opportunity projects as well as the provision of health services and basic infrastructure such as water and electricity fall under S.O. 3.1. The development of economic opportunity projects, primarily for irrigation water and drainage, relied heavily on community mobilization efforts to organize community members who then focused on community needs and priorities. In most of the sites visited, the communities developed health clinic and/or school projects prior to undertaking economic opportunity projects. This "integrated" approach would seem to indicate that separation of community development and economic opportunity from humanitarian oriented health, education, youth, and infrastructure (water, electricity, roads) projects may be counter to the expressed community priorities The economic opportunity projects conducted under SII supported community groups in designing and implementing projects of community-wide interest and impact. These projects were in irrigation and agricultural product processing. Removing these kinds of economic opportunity projects would weaken the community mobilization and community development efforts that have been shown to have an enhanced effect through addressing economic opportunity concerns. The microfinance projects of Save the Children and, in Naxcivan, ADRA on the other hand, have not demonstrated that they contribute significant synergism in support of either community development or health programming. Such projects and grants could be moved to direct USAID management under S.O. 1.3. In making that move, additional regional responsibility will result for S.O. 1.3 since these microfinance projects are currently in Naxcivan and the Central Region of the IDP Belt. In sum, economic opportunity projects that emerge from community mobilization efforts are more appropriate to the humanitarian assistance programming due to the synergy and potential impact on the community at large. This is in contrast to the microfinance projects that were observed which could be managed under S.O. 3.1. Program Effectiveness The Evaluation Team observed many project sites where activities in all three major program areas (community development, health, and economic opportunity) have been successfully attained according to the planned objectives of those communities. Accordingly, to the extent that USAID is aligned with community priorities, USAID objectives and intended results have been achieved in these cases. These successful endeavors also resulted in community organizations that stated that they were sufficiently confident in their own abilities to undertake new projects with and, if necessary, without international donor support. The Evaluation Team noted a major constraint to achievement of community goals. The additional resources available through the USAID sub-grantees had a high impact on community growth, satisfaction and realization of subsequent projects. Inadequately supported efforts failed FINAL REPORT xii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation to achieve their maximum potential. The one case where the USAID financial support was in the range of $10,000 and not sufficient to the several projects that were undertaken, nonetheless, was assistance that was greatly appreciated by the community. They were prepared to move onto other projects despite the fact that the school roof still wasn’t replaced and leaked. The constraints varied from project site to project site. In one case, approval of a community project was denied by the Executive Com, the regional implementation agency of the national government, and required appeal to higher governmental levels. In all other observed cases, support including a mix of encouragement, labor, technical assistance, materials, and direct financial contribution was provided by Municipal and/or Ex Com officials. The Evaluation Team sees an opportunity for USAID to further engage government officials at all levels in future programming. Subgrantee Objectives and Meeting Needs of Targeted Beneficiaries In the cases observed by the Evaluation Team, objectives were largely well defined for each subgrantee. According to beneficiaries (who defined the needs and priorities), the sub-activities were more than meeting their needs and expectations. At every community meeting, the beneficiaries expressed their gratitude for the work accomplished by the subgrantee Partners and thanked the U.S. Government for the support. The Evaluation Team was satisfied that the field sites were representative of results obtained overall by efforts of the subgrantees. Level of Community/Beneficiary Participation The Evaluation Team was impressed with the community and beneficiary participation which was assessed to have been the result of subgrantee community development (mobilization) and community project implementation efforts. In these observed instances, the level of participation was high and more than sufficient to achieve the immediate community objectives. This high level of participation was seen as essential for programmatic sustainability. Enhanced Municipal and Ex Com participation, in some cases greater than Evaluation Team expectations, has the potential to result in increased community project planning and implementation. Participation by IDPs in the community and by women was largely appropriate. The Evaluation Team would encourage a higher proportion of IDP involvement in the IDP belt to support increased acceptance of IDPs in these community settings. Participation by women was fairly high. In only three settings, Camps 4 and 5 in the Southern IDP belt, the World Vision project in Sumgait, and an IDP neighborhood in Barda, was there a 100% proportion of IDP beneficiaries in the observed AHAP projects. In the rest of the twenty or so sites visited, the proportion of IDP beneficiaries ranged from 10 to 25% in the IDP belt. All project beneficiaries were in conflict-affected areas. No IDPs were observed at the sites visited in Naxcivan as IDPs constitute approximately 1% of the Naxcivan total population. FINAL REPORT xiii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Geographic Appropriateness of S.O. 3.1 Activities Community development needs appear nearly universal in Azerbaijan according to the field observations of the Evaluation Team in the areas visited. The Evaluation Team observed ongoing community development needs that could be significantly impacted by the current S.O. 3.1 type activities in the IDP Belt (Central and South), the Sumgait Area, and in Naxcivan, irrespective of the proportion of IDPs in these areas. The geographic appropriateness of Mission activities to follow from the S.O. 3.1 AHAP experience will largely depend upon the extent of resources, both Mission personnel and financial, that can be committed. At the very minimum, the Evaluation Team recommends continuing community development, economic opportunity, and health programming in the existing four regions. Of the four regions, Naxcivan was seen as especially impacted by the lack of access to markets and materials. The Evaluation Team recommends that USAID keep Naxcivan as a high priority area for future programming. The engagement in USAID programming by sub-grantees in addition to ADRA would also encourage the synergism observed by the Evaluation Team in the IDP belt in the Central and South programming regions. These include enhanced INGO partnering, support for broader community development infrastructure priorities through CEEOP and SII-like funding with special attention to potable water, and cluster development. HEALTH PROGRAM Azerbaijan shares with many other parts of the former Soviet Union a heritage of vast hospital over-capacity with more beds than required conjoined with low hospital utilization and excessive lengths of stay in comparison with norms found in Western Europe and the United States. Visits to physicians, clinics and hospitals have been accompanied by an understood requirement for "gratuity" payments. Since independence, in Azerbaijan as in other parts of the NIS, many physicians are unemployed and more have found new employment that make no use of their professional training. While the medical infrastructure – buildings, facilities, equipment – become increasingly outmoded and ever-more deteriorated, the Government of Azerbaijan has been unable to make the investment necessary to restore the former system and appears unwilling thus far to undertake a conscious top-to-bottom review and reformulation of its roles, resources, responsibilities and approach in health programming. Health Program Effectiveness Community perceptions of the quality of health care being received and, for health staffs and community users, the value of professional and public health awareness and education training received through AHAP health programming was without exception seen as substantial. This was especially true with regard to maternal and child health education. Access to potable water at the community taps, made available through several Mercy Corps partners’ projects, was for all an enrichment of great value. Where pharmaceuticals were made locally accessible through revolving drug funds, the value of their accessibility without long travel time and costs was recognized and appreciated. FINAL REPORT xiv USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation In addition, various of the S.O. 3.1 projects represent "new thinking" for the communities served and, even, broader policy discussions and formulation in future. The concepts of revolving drug funds and insurance for health care will be important when, finally, the MOH and others turn seriously to issues of further reforms in the financing of health care, academic training for physicians, primary health care, and system sustainability and reform. The Evaluation Team observed an extensive degree of partnering, cooperation and coordination among and between Mercy Corps’ grantees. Quite simply, no Team member has seen an equivalent or similar level of cooperation and coordination between donor-funded subgrantees. Against these above general conclusions, two exceptions stand out: Pathfinder and AIHA. As noted above, Pathfinder is the only grantee observed whose activities, in their totality, were by intent simply service oriented. Certainly, the beneficiaries with whom the Evaluation Team met expressed substantial appreciation for those services and for those individuals providing them. In several instances observed, caseworkers and beneficiaries seemed to have established close friendships as a result of their interactions over time. Community development, attitudinal change, and a sense of empowerment never arose, however, as spillover benefits. Large parts of the AIHA award are apparently expended in travel to U.S. health centers of grand technology that are achieved through, for Azerbaijan, unimaginably unaffordable costs. The Evaluation Team suggests that a far more modest effort in the training of hospital-based physicians, using curricula and techniques that are closely aligned with and reflective of available Azerbaijani technologies and resources, might well be more practical, efficient and cost-effective. Health Issues and Constraints The key issue affecting all health care delivery-related projects is the future role, policy direction, and funding of the Ministry of Health. Continuing growth of the economy, expected governmental revenues from oil, and a willingness to share the results of growth more equitably across the population are important considerations. The Ministry remains the source of approximately half of all health care expenditures in the country, but these are less than two percent of GDP or approximately $7 per capita. Until and unless this rises, or the current financing system, with regard to both sources and amounts, is very substantially modified, there are simply insufficient funds in the system as a whole to secure significant increases in the general population’s health status. Moreover, currently available MOH resources support a highly inefficient delivery system, with too much spent on hospital care and, both relatively and absolutely, too little spent on primary care, disease prevention and health promotion. With regard to the future of USAID support of health care, two key considerations arise. First, USAID’s resources are insufficient to meet all the useful and good opportunities to assist that are available in Azerbaijan and choices have to be made. Second, community level health services’ projects are demonstrably a useful wedge through which to initiate broader community activities. They, like other individual projects are seldom, by themselves, enough to secure the broader attitudinal, behavioral, and economic spillover necessary to community takeoff. FINAL REPORT xv USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Health Program Sustainability The revolving drug fund sites that were visited seemed to be fully successful, both in Naxcivan and mainland Azerbaijan. The privately operated Fizuli hospital in Babi, built with support to the community by CHF/SII and IMC with initial community mobilization by IRC, is clearly a financial success and its sustainability is assured. The future of other projects, however, is less clear. For example, one of the SII projects visited was a newly built primary healthcare facility, replacing one no longer worth rehabilitating. The building is attractive but virtually empty of the modest equipment needed to render primary care. Several of the partners including IMC, CHF, ADRA offer some form of health insurance, usually on a family subscription basis. The benefit package may vary and several of these also offer fee￾for-service care. IRC offers loans for essential health services beyond the immediate capacity of patients to pay. Take-up rates are highly variable across partners and sites. Health insurance is a new concept to all the communities and the fission that flows from exposure to new ideas is highly desirable. But whether or not “premiums” can be set high enough to cover all relevant costs and sufficient participation rates can be maintained are as yet by no means clear. The Evaluation Team noted that financial sustainability of many of these projects and their approaches to cost recovery may ultimately depend upon decisions by and about the Ministry of Health. Will private practice be allowed to blossom outside Baku? What will be the role of district head doctors and other officials in sanctioning benefits and cost recovery approaches? Will the MOH work with others and pay its “fair share”? What level of financial support will the MOH receive from government income? Health Policy and Program Recommendations Health is an appropriate part of S.O. 3.1 and the Evaluation Team recommends that it remain there. At the observed project sites it was demonstrated that health can be an appropriate fulcrum on which to lever broader community activities and development. It is less clear, however, that its focus should be necessarily limited to the IDP Belt since needs for improved health care seem to extend clear across much of the country. However, the Evaluation Team recommends that there should be a conscious decision to concentrate health and other S.O. 3.1 funds in a limited number of communities to develop over time a critical mass of support for attitude and behavior changes necessary to achieving community and economic development. The IDP Belt, urban concentrations of conflict-affected persons, and Naxcivan appear to be the areas of greatest need. The Evaluation Team, along with USAID/Azerbaijan, recognizes the difficulties associated with working with the Ministry of Health. Nonetheless health, sustainability, and viability, are dependent upon MOH decisions and future health program decisions may be better if outside assistance can be brought to bear. Moreover, in their day-to-day activities, the INGO partners of Mercy Corps not only must deal with ExCom officials, municipality leaders, but also with district hospital head doctors and Ministry of Health officials. FINAL REPORT xvi USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ECONOMIC OPPORTUNITY The Partner organizations under a total of seven subgrants are engaged or have engaged in economic opportunity programming under the AHAP umbrella. ACDI/VOCA conducted a program of Community Economic Recovery Groups (CERGs) that has been extended, removed from the AHAP umbrella, and placed into S.O. 1.3, Economic Development, to be managed directly by USAID/Azerbaijan personnel. CHF, under a single subgrant, implemented the Social Investment Initiative (SII) in three of the four USAID conflict-affected areas and is the sole subgrant to be conducted in more than one impacted area. Save the Children and CHF are the Partner organizations with two separate subgrants for economic opportunity. ACDI/VOCA conducted a program of Community Economic Recovery Groups (CERGs) that has been extended and removed from the AHAP umbrella. The three remaining AHAP Partners, International Medical Corps (IMC), Pathfinder, and World Vision are not directly involved in economic opportunity. IMC is conducting cost recovery components for health services which do have economic ramifications. Loan projects conducted by SC in the Central region of the IDP belt and by ADRA in Naxcivan appeared to have supported borrower enterprises with relatively few failures as estimated by the Evaluation Team (an estimated of 10% of loans did not achieve the borrowers objectives). Repayment rates were at approximately 98%. The conclusion from Evaluation Team visits to these Partner selected sites is that the loan projects were working as planned. Three focus groups, composed of persons with no or very little contact with AHAP programming, were conducted. Two were in the Central region and one was in Naxcivan. Individuals in these groups, in addition to identifying community needs, stressed the lack of economic activity in their areas. The communities visited in the urban area near Baku were noticeably in need of additional economic activity. The obvious conclusion from these focus group sessions and by observation in the urban target area is that economic opportunity projects are widely needed. The economic opportunity projects, conducted under CHF SII, supported community groups in designing and implementing several projects of community-wide interest and impact. These projects were in irrigation and agricultural product processing. Removing these kinds of economic opportunity projects would weaken the community mobilization and community development efforts that have been shown to have an enhanced effect through addressing economic opportunity concerns. The Evaluation Team recommends that such community-wide economic opportunity projects remain within the revised S.O. 3.1 programming. The microfinance projects of ADRA and Save the Children, on the other hand, have not demonstrated that they contribute significant synergism in support of either community development or health programming. Such projects and grants could be moved to direct USAID management under S.O. 1.3. In making that move, additional regional responsibility will result for S.O. 1.3 since these microfinance projects are currently in Naxcivan and the Central Region of the IDP Belt. FINAL REPORT xvii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation COMMUNITY DEVELOPMENT The Evaluation Team found that all of the subgrantee programs affected the communities in which they operated in one or more ways. Of the Partner organizations, IRC appeared to have the strongest and most coherent program of community mobilization and development. IRC utilizes a proven participatory training process that has been improved and applied in a variety of other IRC country settings. The IRC community mobilization begins with a four-day intensive program for up to 150 members of a community or collection of communities. Subsequent training is provided and support is given for microprojects that in subsequent cycles can be as large as $5,000 with IRC financial support. Several of these community groups realized their larger community priorities through linkages with CHF. The CHF program provided grants in the $20,000 range that allowed these communities to develop substantive and quality community projects. The Evaluation Team recommends the inclusion of economic opportunity as important to Community Development in those instances where the majority of the community population would directly benefit. Larger funding for community projects available through CHF was observed to have resulted in quality efforts, an increase in community pride, and a strong sense of community achievement. The Evaluation Team recommends that future community programming be structured to allow community project funding in the $15,000 to $25,000 range. Other Partner organization project efforts in Azerbaijan could benefit from the application of the community mobilization/development methodology of IRC. While participatory methodologies were applied by sub-granting partners o one degree or another, the structured approach of IRC was seen to result in strong community groups that accepted the responsibility for planning and taking actions to improve their communities with or without international donor support. The Evaluation Team recommends that current and future community development efforts require Partner staff training to the extent that similar support can be provided to other community groups in the planning and implementation of the various AHAP approaches to community development, health, and economic opportunity projects. AHAP UMBRELLA PROGRAM The MCI operation of the AHAP umbrella meets or exceeds the mandate to Mercy Corps set forth by USAID in the original Cooperative Agreement. Every grant-funded project, regardless of its sectoral focus, apparently has as its center a community organization element that seeks to build sustainable capacity in target communities. This approach responds to IR 3.1.1.2- Communities Organized to Address Self-Defined Needs. It is also one of the key ways of achieving IR 3.1.1-Vulnerable Communities Better Able to Meet Their Own Needs. The AHAP umbrella has been extremely effective in quickly mobilizing subgrantee Partners and implementing appropriate programming. AHAP I RFAs were issued in March 1998, one month after Mercy Corps was awarded the agreement to manage the program. The second round of RFAs was issued for AHAP II in a similarly timely fashion. FINAL REPORT xviii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Umbrella Effectiveness, Innovations and Issues The most striking and most valuable aspect of the AHAP umbrella—and of Mercy Corps’ management is the atmosphere of collaboration and cooperation that exists among the Partner organizations. It rarely is found to such a degree and greatly increases the impact and effectiveness of the AHAP Partnership. This collaboration was demonstrated in many of the AHAP project sites in the IDP Belt where two and as many as three Partners were engaged in the same community to develop and implement complementary programs. This level of integration of partner activities is an innovation of the Mercy Corps umbrella. The Mercy Corps umbrella has been successful in bringing a wide range of experienced and committed PVOs to address the program areas of S.O. 3.1. The original approach that Mercy Corps has taken is that, from the outset, monitoring was on a geographic basis rather than on grantee by grantee. In effect the separate programs administered by the subgrantees were integrated. This integration was seen by the Evaluation Team as a major factor that contributed to the results achieved by the overall umbrella program. AHAP technical assistance has not always been effective in practice and is viewed by some as redundant in light of the experience and technical expertise of grantees, nor is it a part of the original USAID mandate to Mercy Corps. The AHAP Partners are among the most experienced and respected organizations in their fields. The Evaluation Team observed that partner organizations continually improved their field projects according to real project situations and opportunities. Accordingly, some of Mercy Corps’ resources now committed to monitoring and technical assistance could be redeployed to coordination with USAID which would have greater payoff in terms of programmatic quality. The area in which Mercy Corps has been least successful is creation of trust between grantees and itself. The fact that Mercy Corps is so central to grant award decision-making raises some serious questions of conflict-of-interest. Babi Hospital, in Fizuli District, provides an example of community self-help and the high level of coordination and cooperation between AHAP partners and is one of those projects when everything went “right.” IRC started the ball rolling with a community mobilization/development effort that resulted in a Village Council and the building of a small one room community center. Two surgeon brothers who had worked in an area army hospital during the war with Armenia returned again to practice to there in 1996 after the Armenians retreated from the district. The community of Babi made an old store available which the doctors began to convert into private practice medical facility using their own resources but it quickly proved too small and lacked water and appropriate sanitation facilities. With help from IMC, a six member Community Health Management Committee (CHMC) was formed. With a little financial support from IMC, a new hospital was begun by the community with its own funds and labor but resources soon ran out. CHF awarded the community a $20,000 CEEOP grant, matched by the community with $9,800 in cash and $3,000 in kind and an IMC contribution of $3,000. The completed hospital, with potable water, sterilization equipment, medical waste disposal facility, an operating theater, recovery room, wards and an ambulatory primary care clinic received more than $100,000 worth of medicines, supplies and equipment from IMC and the State Department, and IMC has provided training to the medical staff. It is the only MOH licensed private facility of its kind outside of Baku. FINAL REPORT xix USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation S.O. 3.1 Strategy Development and Implementation According to Evaluation Team interviews, Mercy Corps has had wide latitude in determining the particulars of how the USAID’s S.O. 3.1. Strategy is developed and implemented. The involvement by USAID / Azerbaijan personnel has primarily consisted of review and approvals of RFAs and subgrant awards. The relatively small role of USAID/Azerbaijan in designing and managing the evolving program strategy seems a missed opportunity, both from the standpoint of fine-tuning its work in S.O. 3.1 and lessons that would inform its larger assistance program in Azerbaijan. Without impacting the management efficiencies inherent in the umbrella, USAID could profit from more active involvement in determining program direction and evolution. MCI Monitoring and Compliance The monitoring function is the central vehicle for the Mercy Corps’ approach to managing the AHAP umbrella. It is combined with a technical assistance function, which it should be noted is not a part of the USAID mandate to Mercy Corps. Mercy Corps has moved toward an integrated approach in AHAP, attempting to bring the full panoply of services in economic opportunity, health and community development to bear on target communities. Reflecting this approach, it has adopted an integrated approach to monitoring, in which a team of Mercy Corps personnel visits grantees for up to five days. A number of subgrantees indicated that Mercy Corps monitoring is burdensome because of its frequency, although recently reduced to quarterly from every two months—combined with the depth and duration of visits as well as required reports. They pointed to time consumed not only by the visits themselves but the advance preparation required of them. The Evaluation Team also recommends that the number and type of program indicators be reviewed and reduced. On balance it seems that Mercy Corps’ monitoring is excessive and that the frequency should be reduced to semi-annual. In general, the compliance function (helping subgrantees to comply with USAID policies, procedures, and regulations) is carried out well. Few grantees indicated concerns about lack of information or misinformation nor did USAID indicate major concerns about Mercy Corps’ ability to secure satisfactory compliance. Umbrella Mechanism as an Implementing Vehicle To date, the umbrella has been an extremely effective management tool for USAID/Azerbaijan, and one USAID individual was assigned the responsibility for internal management. The umbrella has yielded a high quality and relevant program. It has had the subsidiary benefit of allowing for a broad mix of organizations to operate in Azerbaijan through its effective process of subgrantee RFAs, more so than may have been possible had USAID been managing all of the S.O. 3.1 activities directly. In addition, and very importantly, the umbrella has fostered a unique level of coordination. FINAL REPORT xx USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation It is doubtful that USAID’s costs would be lower in managing the program activities directly. To do so, especially with the numbers of implementers involved, would require at least one additional expatriate staff person in addition to the Humanitarian Assistance Officer now on board and several additional local staff persons. It also would require two or three additional financial management personnel and attendant costs of increased transportation and office space. There could be internal issues of Mission operations versus program budgets as well. The choice for USAID on using an umbrella manager or adding the necessary technical, administrative, and contracting personnel to Mission staff will be a function of internal priorities. Where do the activities now carried out under S.O. 3.1 fit in the larger scheme of USAID objectives and do they support other more important objectives? UMBRELLA DEVELOPMENT/IMPLEMENTATION RECOMMENDATIONS All things being equal, a continued umbrella is preferable to direct management by USAID. The major reasons are the more intensive oversight and monitoring that an umbrella can provide and the richer mix of implementing organizations it affords. The ultimate decision of USAID/Azerbaijan will of course depend on considerations about its overall assistance program in Azerbaijan, available resources, and internal management capacity. On balance, a PVO umbrella manager is the logical manager for a future umbrella, both for reasons of cost and of culture. The shared cultures of the PVO Mercy Corps and its PVO subgrantees have yielded positive results that a for-profit umbrella manager would be unlikely to duplicate easily. USAID/Azerbaijan and its S.O. 3.1 programs will benefit from the planned development of a new USAID strategy that reflects the current development environment in Azerbaijan. The PMP that is developed for the new strategy should focus on qualitative information relevant to the fundamental objectives the S.O. 3.1 program is aimed at achieving and should have many fewer indicators than the 150 output-related indicators in the current plan Umbrella Review and Program Strategy If the umbrella mechanism is continued, USAID/Azerbaijan should institute a system of formal review of program progress with the umbrella manager. The review would address issues such as achievements to date, possible refinements, lessons learned and future planning. The review should be conducted semi-annually, using the umbrella manager’s report as its basis. Staffing and Personnel Resources In a future umbrella, staffing should be reduced by approximately 10 persons. The umbrella itself should have no more than three expatriates (Chief of Party, Program Director and Chief Financial Officer). De-emphasis on internal technical assistance and reduced monitoring frequency would enable these reductions. FINAL REPORT xxi USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation To fill the potential gap occasioned by a reduction in full-time internal technical assistance capacity, the umbrella manager should budget funds for short-term technical assistance, as required by technical requirements of umbrella administration. The umbrella manager should maintain a database of appropriate consultants with specialized areas of expertise should grantees require assistance in locating the required skills. Similarly, grantees should be encouraged to budget for consulting in their respective budgets. MISSION STRATEGY DEVELOPMENT The Evaluation Team recommends that USAID / Azerbaijan consider the development of a structure of strategic objectives that recognizes the accomplishments of an evolving program in the conflict-affected areas. Basic humanitarian needs of food and shelter are being provided by other donors and, to some extent, by the GOAZ. While there is substantial need for economic opportunity programming, the general economy, even in Naxcivan, has been improving over recent years as shown in World Bank economic indicators for Azerbaijan over the past five years. Gross domestic product has increased at high rates and the inflation rate has been substantially decreased. USAID/Azerbaijan has already initiated the process of defining strategic objectives to develop a strategy statement that reflects the improving humanitarian situation in Azerbaijan and the impact of previous and current USAID programming. It is recommended that this strategy statement continue to have a humanitarian oriented context. It is recommended that since basic human needs have been met at some level, the revised S.O. reflect the phasing to broad-based community and economic development to targeted communities of greater need and potential as contrasted with S.O. 1.3 target areas of greater opportunity. The required set of RFAs, RFPs and extensions as appropriate should be completed over the next seven to nine months to allow an orderly program development and minimum disruption of related USAID current projects by January, 2004. The revised strategy should also take advantage of programmatic insights that may be developed from the Cluster Development subgrant extensions of World Vision and IRC. In addition, IRC will be implementing a Municipal Government oriented subgrant extension that may provide relevant findings for S.O. 3.1 strategy development. The Evaluation Team also recommends that the revised strategy consider approaches to strengthen the local NGO community. Government Interaction The Evaluation Team found that opportunities exist for direct engagement with local governmental officials to deal with concrete issues that affect the well being of the S.O. 3.1 target population. Previously limited by Section 907, simple issues can now be addressed and include equipping refurbished clinics and schools or more complex matters like acknowledging and perhaps even encouraging official adoption of cost recovery and health insurance mechanisms. FINAL REPORT xxii USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Several partner informants made the case for concerted advocacy beginning at the local level and moving upward to the national level on such matters. AHAP partners at the local level, AHAP at the regional and national level, and USAID at the highest echelons provide a structure for coordinated advocacy. The Evaluation Team makes these suggestions in the full knowledge of the constraints of an autocratic and centralized government system and with the trust that the new latitude afforded by the 907 waiver offers an important new opportunity for development programming in Azerbaijan. Sustainability The sustainability of S.O. 3.1 activities is uncertain. Most of what the Evaluation Team saw was less than one year old so that it is not clear that the initial community mobilization efforts will be maintained. Local NGO capacity-building is the usual strategy for ensuring sustainability, but this approach is hampered in the current enabling environment of Azerbaijan. Now, however, there is the opportunity to go further in an admittedly hostile environment to address the formalizing of local NGOs. An initial step can be the engaging of an organization like the International Center for Not-for-Profit Law (ICNL) to help devise a strategy to foster a more enlightened NGO enabling environment as a part of USAID’s Democracy and Governance initiatives. FINAL REPORT 1 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 1 INTRODUCTION This evaluation was conducted with field visits to project sites and interviews in Baku during the period October 7 through 25, 2002. In accordance with the Statement of Work, the evaluation examined USAID/Azerbaijan’s Strategic Objective 3.1, Reduced Human Suffering in Conflict￾Affected Areas, with emphasis on the Azerbaijan Humanitarian Assistance Program (AHAP). The evaluation looked at AHAP both as a set of activities designed to achieve the strategic objective and as a mechanism for managing those activities. The evaluation includes recommendations on best ways to ensure the most efficient delivery of program support that address the needs of the target beneficiaries. Based on findings, the evaluation also provides specific strategic and programmatic recommendations for USAID’s further assistance in this sector. 1.1 STRATEGIC OBJECTIVE BACKGROUND Although USAID commenced activities in Azerbaijan in 1992, USAID/Caucasus/Azerbaijan’s first formal program strategy started in FY 2000 and was scheduled to end in FY 2003. The program strategy has since been extended to January 2004. One of the four Strategic Objectives of this current strategy is S.O. 3.1, Reduced Human Suffering in Conflict-Affected Areas. The USAID budget for Azerbaijan is $28.3 million for FY 2002 ending September 30, 2002. Of the total USAID budget, $10 million or 35% of the total is under S.O. 3.1. The major vehicle for providing assistance under Strategic Objective 3.1 is the Azerbaijan Humanitarian Assistance Program (AHAP) which is implemented through a cooperative agreement with Mercy Corps International (MCI). AHAP commenced in January 1998 and, as amended, is a six-year activity with a funding level of $45 million that ends in January, 2004, an approximate 14 months from the completion of this evaluation. The three-year USAID/Azerbaijan strategy is in the process of being updated to reflect programmatic changes made possible by the January 2002 Presidential waiver of Section 907 of the FREEDOM Support Act (FSA). Although humanitarian assistance was exempted from Section 907 in 1999, AHAP already consisted of predominantly community development and economic opportunity programs at that time and so continued to be constrained by the application of Section 907 in many instances. Under the waiver of Section 907, activities under S.O. 3.1 have been able to work openly with the Government of Azerbaijan (GOAJ) and are enabled to fund projects within the Government. USAID/Azerbaijan, under the waiver of Section 907, will be able to work with the Government of Azerbaijan (GOAJ) if such work will support and promote program objectives. The AHAP is the primary vehicle for providing assistance under S.O. 3.1 and was a successor to a Caucasus-wide umbrella mechanism implemented by Save the Children from 1993 to 1998. FINAL REPORT 2 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Significant budgets were provided to address the urgent humanitarian crisis in the South Caucasus at a time when USAID staffing in the region was minimal. This was the primary reason that USAID selected umbrella mechanisms to implement a significant portion of the program. Currently, all AHAP activities are implemented through the award of sub-grants to international NGOs (mostly as part of competitive processes) and are concentrated in four geographic regions: Central and South – together comprising the so-called "IDP Belt;” Naxicivan, and the Baku urban area. Activities are funded in three major program areas -- community development, health, and economic opportunity. Program delivery by sub-grantees is scheduled to end in October 2003. USAID/Azerbaijan health sector activities outside of AHAP consist of the Health Partnership program (administered by the American International Health Alliance, AIHA) and a grant to UNICEF for primary health care. The only other non-health activities for internally displaced persons (IDPs) are two grants to UNDP for NGO development (related to the World Bank funded Social Fund for the Development of IDPs, SFDI) and for demining. While work implemented under these activities were considered when assessing the effectiveness in achieving results defined under the strategic objective, none of these activities were directly evaluated under the Evaluation Team scope of work. 1.2 EVALUATION PURPOSE The purpose of this evaluation is to review S.O. 3.1, its associated activities, and the performance of AHAP as a management mechanism. The findings and recommendations will inform Mission management and the implementing partners on how effective the current program has been in meeting the strategic objective and where and how possible adjustments could be made to improve program implementation and impact. The Statement of Work included specific evaluation questions based on the review of S.O. 3.1 strategy and objectives, S.O. 3.1 activities, and the AHAP mechanism in accordance with past performance and future prospects. The Evaluation Team discussion on these questions is presented in subsection 7.1 of Section 7, Findings and Recommendations. 1.3 COUNTRY CONTEXT Azerbaijan has been inhabited for at least 3000 years. Archaeologists suggest that 6000 years ago, people had constructed structures around the gas vents of mud volcanoes and the spontaneous ignited methane fires found on the Absheron Peninsula near modern day Baku. Bronze Age settlements of the beginning of the first millennium BC have been found in and around Baku. Scythians settled in the area in the 9th century BC, followed by the Medes, followers of Zoroastrianism, the first major monotheistic religion. The prophet of Zoroastrianism, Zarathustran, may have been inspired by the spontaneous flames around Baku where he spent a considerable part of his life. The Archaemenid Persians took over half the country 200 years later, only to be conquered by the Greeks and Alexander the Great in 331 BC. Alexander called the area FINAL REPORT 3 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Atropatena (or in Persian Aturpatan - “protected by fire”) from which the name Azerbaijan is derived.1 Beginning in the 1st century AD, the general area in which Azerbaijan is located came under the control of the Romans. After the 3rd century the Persians returned to dominate the area. Islam entered the region with the Arabs in the early 8th Century and the Arabs had control until being displaced by the Seljuk Turks in the 11th Century. The Turks from Central Asia progressively had come to populate Azerbaijan and form the ethnic and linguistic foundation of modern Azerbaijan. Beginning around 1050, the country enjoyed a cultural renaissance that witnessed many of its greatest architectural, artistic, and literary achievements. For a seven hundred year period, the Ottoman Turks and Saffavid Persia struggled for control of Azerbaijan. In 1722, Peter the Great of Russia occupied much of Azerbaijan and the Caspian Coast. This Russian incursion was displaced soon by an Afghan descended Persian ruler. A three-way struggle between Russia, Turkey and Persia ended in 1813, when Russia and Persia divided Azerbaijan along the Aras River. Subsequent geographic divisions of land and peoples between Iran and Russia have resulted in a modern day Azeri speaking population in Iran which is estimated to be as much as triple2 the current Azerbaijan population of 8 million. Azerbaijan had a brief taste of independence between 1918 and 1920, but was combined by the Soviet Union into a “federated Republic” with Armenia and Georgia in 1922. The Soviets dissolved the federated Republic in 1936 but held the three Republics within its orbit. After the brief Soviet occupation of northern Iran during WWII, the Iranian government crushed the nascent independence movement that had been started there by ethnic Azerbaijanis.3 In 1991 with the collapse of the Soviet Union, Azerbaijan regained independence. Naxcivan, the Western portion of Azerbaijan that is separated from the “mainland” and is bordered by Armenia on the North, Iran on the South, and Turkey at the Western tip, upstaged even Lithuania to be the first Soviet Republic to declare independence. The 1988 to 1993 era is overshadowed however by the Nagorno-Karabakh dispute and war with Armenia. 1.3.1 Armenian-Azerbaijani Conflict and the Peace Process Armenians living in Nagorno-Karabakh had long resented Azerbaijani rule; the conflict erupted in 1988 and escalated after Azerbaijan's independence in October 1991. Armenian attacks on Azeri citizens in the region prompted repeated attacks from Azerbaijani forces, leading to a string of defeats for the Azerbaijanis and the resignation of two presidents. By 1993, the conflict had created thousands of casualties and estimates are as high as one million refugees/internally displaced persons. The June, 2002 UNHCR Report/Statistical Overview lists 572,541 Internally Displaced, 51,649 Various Others, 3,376 Asylum Seekers, 287 Refugees, and 27 Returned Refugees for 627,790 Total Persons of Concern to UNHCR. 1 Mark Elliott, Azerbaijan, with excursions to Georgia, Trailblazer Publications, UK, 2001, p.38 2 At a press conference given in 2002, Ilham Aliyev- Head of Azerbaijan delegation to the Council of Europe and member of the Azerbaijan Parliament estimated the Azeri population in Iran to be about 30 million. 3 Cornell, Svante, “Undeclared War” in Journal of South Asian and Middle Eastern Studies, vol 20, no 4, 1997 found on http://www-scf.usc.edu/baguirov/azeri/svante_cornell.html FINAL REPORT 4 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation A cease-fire in 1994, following the retaking of a portion of the Armenian occupied area under the leadership of President Aliev stemmed the worst of hostilities. In 1999 Nagorno-Karabakh declared itself a Republic and, although not officially recognized by any country in the world today, showed little interest in giving back any territory to Azerbaijan, including the narrow strip of Azeri land connecting Karabakh with Armenia proper. Active hostilities ended in 1994 with the support of an OSCE brokered cease-fire and the fourteen￾year-old Armenian-Azerbaijani conflict has yet to be resolved. The OSCE Minsk Group has worked on a negotiated settlement for ten years and there has been little progress made. It is generally agreed that the negotiations are at a deadlock. Under the initiative of the United States, France and Russia (the co-chairs of the Minsk Group), the Special Representatives of the respective governments met in Prague from May 8 through May 15, 2002 to discuss negotiations towards a settlement. No substantive progress was made at these meetings.4 The reasons for current situation are apparently not due to a lack of effort on the part of the OSCE or the Minsk. Three alternative peace proposals have been submitted to Armenians and Azerbaijanis and more than a dozen meetings have been sponsored. The three peace proposals were rejected and all meetings have failed. The Azerbaijanis see themselves as the aggrieved party in the conflict. Armenians have captured 20% of Azerbaijani territory and at least 600,000 Azeris were expelled from the Karabakh area. The Azerbaijanis mistrust Russia because it backed Armenian troops during the armed stage of the conflict, and they distrust the Armenian lobbies in the US and France. The Minsk Group is seen as looking for economic solutions in disregard of the OSCE and UN principles of territorial integrity. A key Armenian negotiating point is economic cooperation rather than territorial integrity. The Minsk Group is seen as misreading the Azerbaijani position that considers territorial integrity as more important. The basic public positions of both parties appear to as follows: Armenians feel that there should be self-determination for the territories of Nargorno-Karabakh while Azerbaijanis want territorial integrity.5 The situation has been at a standstill since the spring of 2002. It has been contended from the Azerbaijani side that a modification in the mediation style of the Minsk Group could support subsequent negotiations. A fourth change of format is underway, at the time of this report, in which the Deputy Foreign Ministers for each country meet to discuss ways to resolve issues. 1.3.2 Governmental Structure Azerbaijan is a Republic with a strong presidential form of government. Major factors are the health and age of the charismatic and widely respected third Azerbaijani President, Heydar Aliev. He was reelected in October 1998 in a controversial election marred by numerous and serious irregularities, violations of the election law, and lack of transparency in the vote counting process at the district and national levels--and by the ruling New Azerbaijan Party that he leads. Aliev is 78 years old and is apparently suffering from heart and other ailments. He has affirmed that he 4 Greene, Richard Allen, “Armenia/Azerbaijan: As Minsk Group marks 10 years, Karabakh peace appears more elusive than ever” in Radio Free Europe/Radio Liberty, 25 March 2002. http://www.reliefweb.int/w/rwb.nsf6686f45896fl5dbc852567ae00530132/ 5 Ismailzade, Fariz, “OSCE Minsk Group: Is There Space for Improvement?” on Central Asia Caucasus Analyst, website, June 19, 2002, http://www.cacianalyst.org/2002-06-19/20020619_Minsk_group.html FINAL REPORT 5 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation does not want to surrender the claim to Nagorno-Karabakh or to grant autonomy to the enclave. Aliev has made several moves to transfer power to his son. The Constitution states that the Azerbaijan system of government is based on a division of powers among a strong presidency, a legislature with the power to approve the budget and impeach the President, and an independent judiciary. Parliamentary elections in November 2000 and in January were seen in international circles as marked by numerous and serious flaws, particularly in the vote counting process. Parliamentary by-elections held in November in Tovuz and Adjabedi to fill vacant parliament seats also were considered marred by election fraud and ballot box stuffing. Opposition members make up only a small minority of parliament's members. The judiciary has failed to remedy election irregularities, and as a result some domestic groups regard the Parliament as illegitimate. The judiciary does not function independently of the executive branch and is corrupt and inefficient.6 By-elections will be held in November 2000 to fill approximately 40% of the 5,000 Municipality positions that were elected in the December 1999 elections. In addition to the locally elected positions of chairman and members of the municipal council, regional administration by the national government is through appointed Executive Committee (Ex Com) personnel. President Aliev's ruling New Azerbaijan Party triumphed in the November 1995 parliamentary elections, October 1998 presidential elections, December 1999 municipal elections and November 2000 parliamentary elections. The Council of Europe (Azerbaijan is a member) and the Organization for Security and Cooperation in Europe (OSCE) are interested in Azerbaijan’s proven usefulness as an alternative source of energy and seek increased stability and an improved human rights record in the region. 1.3.3 Economy During the initial period of Russian rule in the 19th Century, Azerbaijan's economy grew in relation to Russia's. Azerbaijan provided the later Soviet Union with crude oil, chemicals, textiles, food, and wine. While its days as a Soviet supplier diminished its petroleum supply, Azerbaijan remained a healthy producer of crude oil and textiles throughout the 20th century. The Caspian region is reputed to hold about 100 billion barrels of oil and about as much natural gas, and Azerbaijan has laid claim too much of it. Azerbaijan's State Oil Company made agreements with international firms for exploration and production during the 1990s. As a result, Baku experienced a boom with the first oil coming ashore in 1997. Substantive oil revenues, however, are expected to be realized in the future. According to the U.S. Department Human Rights Report for 2001, the Government continues to affirm its commitment to development of a market economy, but widespread corruption and patronage reduce competition, and the absence of essential reforms has limited economic development outside the oil and gas sector, which accounts for over 90 percent of the country's 6 Bureau of Democracy, Human Rights, and Labor, Country Reports on Human Rights Practices-2001, U.S. Department of State, Washington, DC, March 4, 2002 FINAL REPORT 6 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation export revenues. Approximately 98 percent of the country's farmland is privatized, but commercial agriculture remains weak, and subsistence farming dominates the rural economy. Much of the labor force is employed in the state sector where wages are low with professionals such as medical doctors receiving salaries as low as $10 per month. The overall economic situation of the average citizen remains tenuous. Although a growing middle class has emerged in Baku, over 60 percent of the population live in poverty, according to World Bank statistics. Severe disparities of income have emerged that are attributed partly to patronage and corruption. The economy of Azerbaijan has enjoyed remarkable growth over the past decade. Gross domestic product (GDP) has risen rapidly, as has GDP per capita. Balance of trade has moved from strongly negative to modestly positive. These positive changes are reflected in the summary table below. Key Economic Indicators of Azerbaijan (World Bank Development Indicators) 1997 1998 1999 2000 2001 Population (millions) 7.838 7.913 7.983 8.049 8.114 Current GDP (US $, in billions) $3.962$4.446 $4.581 $5.269 $5.692 GDP/capita (US $) $505$562 $574 $655 $702 Inflation (%) 9.2 -0.1 2.2 12.4 3.1 Exports (% of GDP) 29.0 22.7 28.0 40.7 41.0 Imports (% of GDP) 53.0 54.4 41.9 38.4 37.4 Heavy dependence upon external donor contributions for feeding of IDPs and refugees is mostly a thing of the past. Oil revenues are climbing, a new oil pipeline to the West is under construction, and imports of electricity are beginning to decline. On visual examination of the Evaluation Team, the housing stock is showing some improvement and the government has spent millions on new residences in preparation for moving IDPs and refugees from makeshift camps to better housing. Yet for all these signs of improvement, there remain very significant misdistributions in resources and a substantial lack of income equity. Approximately 60 percent of the population lies below the poverty line. Unemployment and underemployment are both serious problems throughout the FINAL REPORT 7 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation country. Numbers of IDPs and refugees, a decade after the war between Armenia and Azerbaijan, are still housed in boxcars on railway sidings. Large portions, especially of the rural population, lack convenient and proximate access to potable water, quality education and health care and corruption of various sorts remains rife. Oil extraction and delivery, once thought to be the near-term salvation of the Azerbaijan economy, are accomplished at high cost compared with alternatives sources. Market demand is closely tied to world prices; if these drop significantly below the current $25-30 per barrel, Azeri oil production may no longer prove profitable. Moreover, recent exploratory drilling in the Azerbaijan portion of the Caspian has so far failed to prove the large reserves earlier thought to exist. Even if new oil finds of substantial magnitude are found, there remains some substantial question to what extent revenues will significantly increase the lot of much of the population, which remains largely dependent upon agricultural production. And the agricultural sector suffers from a variety of ills. Technology and equipment are too often outmoded or non-existent; traditional export markets for grains are no longer easily accessed; significant areas are suffering from salts’ incursion; and irrigation systems and water supply are both in peril. In addition, much of the country’s most fertile agricultural land and sources of irrigation waters lie in Nagorno-Karabakh and the surrounding areas now held by Armenia. Naxcivan is cut off economically from the Azerbaijan mainland as a result of the severing of the rail link by Armenia. 1.4 EVALUATION APPROACH AND ACTIVITIES The MSI/MetaMetrics Evaluation Team composed of Leo T. Surla, Jr., Christopher Bladen, and David A. Smith reviewed available USAID program documentation prior to departure for Azerbaijan. They participated in a Team Planning Meeting held at the offices of MSI in Washington, DC and arrived in Azerbaijan on October 5, 2002. The first meetings, on October 7, 2002, were with the Mission personnel: William D. McKinney, Valerie Ibaan, and Gulnara Rahimova. An orientation for the Evaluation Team was conducted by Mercy Corps, the AHAP umbrella organization, that afternoon and interviews were conducted with subgrantee personnel in Baku over the next three days. A meeting was held with Mr. McKinney and U.S. Ambassador Ross Wilson. Field visits were conducted from October 11 through October 22, 2002. On October 24, 2002, an Interim Report was delivered to the Mission and an interim briefing was conducted. The analysis of the Evaluation Team of the humanitarian assistance programming is based on reviewed documentation, interviews with key cooperative agreement and subgrantee personnel in Baku as well as with USAID/Embassy personnel and the Azerbaijan Deputy Prime Minister for Humanitarian Assistance, and field visits to approximately twenty-five health, community development, and economic opportunity projects. The site visits included interviews with community groups organized under the activities of the AMAP Partners and with focus groups with no or little contact with the AHAP programming. Field visits were conducted in the four USAID regions designated for AMAP programs: the Sumgait area near Baku, the IDP zone of Central and Southern Azerbaijan, and the Autonomous Region of Naxcivan. All of these areas have been impacted by the Armenian-Azerbaijani Conflict. On these visits, the Evaluation Team FINAL REPORT 8 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation was accompanied by key S.O. 3.1 program personnel of USAID/Azerbaijan: Valerie Ibaan and Gulnara Rahimova. The initial findings, conclusions, and recommendations of the Interim Report were refined after briefings with USAID/Azerbaijan personnel. Additional Evaluation Team analysis relative to improving AHAP program implementation and impact was conducted and incorporated into the final draft evaluation report which was delivered on October 31, 2002. Two briefing sessions were held with Mission personnel prior to the departure of the Evaluation Team from Baku. Comments from USAID/Azerbaijan were received on November 19, 2002. The evaluation report was finalized on December 7, 2002. A project briefing with personnel at USAID/Washington was scheduled for December 9, 2002. FINAL REPORT 9 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 2 HEALTH PROGRAMS Azerbaijan shares with many other parts of the former Soviet Union a heritage of vast hospital over-capacity. Hospitals contain more beds than required. There is low hospital utilization and excessive lengths of stay in comparison with norms found in Western Europe and the United States. There remains, more than ten years after independence following the breakup of the USSR, the shambles of a pyramidal system (the "Semanshko" model). Community level physicians and health points ill equipped to provide effective science-based medical care move patients up to polyclinics and hospitals where most medical care actually occurs as provided by large numbers of specialty physicians. Although inefficient in its construct, that previous system secured high levels of health care and health status for its population: infant and maternal mortality rates were low, life expectancy high, nutrition generally good, and potable water widely available. 2.1 DEVELOPMENTS SINCE INDEPENDENCE The promise of free health care in Azerbaijan after 1991 was a chimera. Visits to physicians, clinics and hospitals have been accompanied by an understood requirement for "envelope" or "gratuity" payments. Now there are official price lists for many services. Physician and other personnel payments by the MOH, however, are inadequate to meet employees’ economic needs and unofficial payments remain substantial, perhaps most notably for abortion services and surgeries. Azerbaijan differs from many of its peers in the newly independent states (NIS) of the Soviet Union in one way: inadequate though Ministry of Health (MOH) payments may be to sustain economic life, those salaries are actually paid and paid on time. Since independence, in Azerbaijan as in other parts of the NIS, many physicians are unemployed and more have found new employment that make no use of their professional training. While the medical infrastructure – buildings, facilities, equipment – become increasingly outmoded and ever￾more deteriorated, Azerbaijan has been unable to make the investment necessary to restore the former system and unwilling thus far to undertake a conscious top-to-bottom review and reformulation of its roles, resources, responsibilities and approach. Not only has there been policy stasis but, worse, the MOH has been widely viewed as perhaps the most corrupt component of government by the nation’s citizenry. And, worst, in many areas health status indicators remain below those of a decade and more ago although there have been improvements from the low point that was reached in the interim. Real health care expenditures by the government have been estimated to have dropped as low as 25 percent of pre-independence levels (Women’s Reproductive Health Survey, Azerbaijan, 2001—preliminary report), and currently represent less than two percent of GDP. Against this background, there are further declines occasioned by the long war between Armenia and Azerbaijan that began in 1989; a truce was not signed until 1994 and there is still no peace accord. That war led to the displacement of eight hundred thousand to a million persons, the internally displaced persons and refugees (IDPs/refugees), found in a wide north-south swath FINAL REPORT 10 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation through the middle of the country. International non-government organizations (INGOs), USAID and other donors have sought to patch together effective health care for the country’s populations that are most at risk. 2.2 PROGRAM/PROJECT DESCRIPTIONS Humanitarian assistance by USAID has been undertaken through Strategic Objective 3.1, and largely through an "umbrella" cooperative agreement, first to Save the Children and for the past five years, to Mercy Corps. Through that mechanism, a variety of INGOs has received the USAID-funded Mercy Corp subgrants to provide a wide range of health-related activities in Azerbaijan. In addition, USAID has directly funded a limited number of additional activities outside of S.O. 3.1. Of the latter, the largest has been to the American International Health Alliance (AIHA). 2.2.1 American International Health Alliance (AIHA) The Evaluation Team did not visit any of AIHA’s trainees or hospital sites and did meet with AIHA’s Azerbaijani director. AIHA serves as an intermediary, "partnering" in-country hospitals and medical staff and management professionals with teaching hospitals, medical schools and academic public health centers in the United States. AIHA has more than 80 projects in the NIS; work in Azerbaijan is among the most recent because of the limits Section 907 has placed upon working with the government of Azerbaijan. AIHA funding in Azerbaijan is supported via an earmark against USAID/Azerbaijan’s budget and is not under USAID/Azerbaijan’s direct management. Unlike most other S.O. 3.1 health activities, the AIHA partnership is intended directly and only to support medical training. There is no intent to directly affect community attitudes and development or behavior outside the medical/health community. Pairs of Azerbaijan and US partners jointly develop the work plan, goals, objectives, activities and indicators that are to guide their relationship. AIHA provides logistical and administrative support, supports travel and related costs for the partners to travel between the two counties for observation and training, communications technology (internet access, remote transmission of x-rays, etc.) for the partners, and provides an equipment budget of $3-$50 thousand per partnership. AIHA partnerships include Mir Gasimov Republican Hospital, partnered with Baylor University, and Marimmov District Health Administration partnered with Portland’s Oregon Science and Health University, and Virginia Commonwealth University (Richmond) partnering with Binagadi Health Administration. The Republican Hospital, located in Baku, is responsible for outpatient services to women care in three camps in the central IDP/refugee belt. The Republican Hospital has received support to develop a neonatal center and to improve the quality of and patient satisfaction with outpatient care. In two district health administration projects, the Azerbaijan team was composed of physicians and nurses, not Azerbaijan administrators who, notionally, would have been expected to participate on the team. AIHA has provided support and travel costs for nurses to participate in its International Nursing Leadership Institute. Azerbaijan nurses who had been participants sought to establish a local NGO, the Republican Nursing Association following their return to Azerbaijan. The Ministry of FINAL REPORT 11 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Justice has failed to register the Association, along with other new local NGOs, for nearly two years. Among other things, registration is required for an organization to be eligible to establish a bank account. In addition to its other activities, AIHA has supported development of treatment protocols by partnerships using evidence-based medicine and funded a breast exam center, to train medical staff in effective breast exam and to build a curriculum for self-examination by women. All relevant activities have included efforts to improve data at the institutional level but AIHA is dubious of their current value given the quality of and interest in data held by the MOH. 2.2.2 International Medical Corps (IMC) International Medical Corps (IMC), internationally, has a history stretching back to 1984 and assistance to Afgan refugees in Pakistan during the war with Russia. In Azerbaijan, IMC has had two sources of support from the U.S. Government. The Department of State, Bureau for Population, Refugees, and Migration has supported its family planning and reproductive health (maternal and child health) program. Through the cooperative agreement with Mercy Corps, USAID supports directly IMC's community based primary health care development program, professional medical training (through a subcontract to the International Rescue Committee), and technical assistance on health infrastructure (through a sub-contract to CHF). In a number of these activities, IMC has secured the assistance of five local NGOs. IMC works in 15 IDP/refugee camps and 33 local villages in six districts. Like several of the other Mercy Corps-funded INGOs, IMC’s community development activities are initiated through a baseline survey of the available primary health care network and examination of the population’s demand for and utilization of health care, both through survey and focus groups. With this information, as necessary, nurse, midwife and physician training are undertaken and essential drugs, supplies and equipment are provided to redirect and reinforce the primary health care service delivery system. Thirty-eight medical clinics have been selected by IMC and of these, 36 have been raised to WHO standards, with ongoing IMC peer monitoring. This usually involves a variety of small-scale projects to assure clean water, appropriate disposal for medical wastes, and other basic facility needs. Concurrently, IMC undertakes community driven evolution and development activities, beginning with community popular selection of a community health management committee (CHMC), responsible for building consensus around identification, prioritization and proposed solutions to the problems it faces. To accomplish these steps, CHMC members are provided training, including the preparation of proposals and fundraising efforts. To support sustainability, each community also establishes a community health fund with an oversight facility management board. Approximately 40 percent of the CMHC members elected have been women. IMC has prepared a variety of health education materials and seeks to assure the inclusion of appropriate health education in community schools. In support of this goal, it has trained teachers as trainers and has developed reproductive health and family planning materials for their and students’ use. FINAL REPORT 12 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation IMC project communities, thanks in substantial part to the close partnering among Mercy Corps￾funded grantees, have been successful in obtaining a wide variety of follow-on projects for infrastructure development and improvement, notably from CHF and IRC. 2.2.3 Adventist Development and Relief Agency (ADRA) ADRA activities are more fully described in the Naxcivan Section 5 of this report, where almost all of their current activities are located. Worth noting here is an effort of national scope, The Women’s Reproductive Health Survey, conducted by the Centers for Disease Control and Prevention (CDC) of the US Department of Health and Human Service under contract to ADRA. This survey, results of which have been delayed for nearly a year but are anticipated for delivery very late in 2002, used 30 specially trained Azeri women as interviewers of a weighted, nationally representative probability sample of 8,246 households in which at least one eligible (age 15-44 years) woman was found. The response rate was 93 percent. The CDC’s Women’s Reproductive Health Survey is the first national survey of contraceptive knowledge and use, fertility, pregnancy, delivery and breastfeeding patterns, and the prevalence of routine gynecological visits, cervical cancer screening and breast self-examination. It is also one of the few activities to date also to bear the imprimatur of the Azerbaijan Ministry of Health. The importance of the survey lies in two facts. First, it will provide data not available elsewhere, largely because of the disinterest of the Ministry in using its own resources to develop reliable information in this area. Second, it is anticipated that results from the CDC study will show infant and maternal mortality rates substantially higher than the "official" MOH data, in part because its findings are not limited to MOH-funded facilities. 2.2.4 Pathfinder International Pathfinder is unique among the health related INGOs supported by USAID through the Mercy Corps umbrella as the only one which has been, until now, exclusively devoted to the direct delivery of services and is the only INGO with entirely local staff. The core of the Pathfinder effort, supported through a $600,000 grant, is educational services to 86,000 persons, many of them IDPs and refugees located in Sumgait, in reproductive health, family planning, breast feeding, childhood immunization, and the prevention of sexually transmitted diseases, including HIV/AIDS. Pathfinder has trained 96 caseworkers, of which 10 are men and 44 are physicians, in community￾based family planning services. The client base of 76,000 women and 10,000 men is regularly contacted by case workers to identify needs and reinforce health education. In its activities, Pathfinder has regularly partnered with a local NGO, the Azerbaijan Women’s Health Center. With help from Mercy Corps, Pathfinder has attempted to address sustainability in its current follow-on project extension through plans, first, to reduce the number of case workers by ten and retrain the remainder as trainers of trainers, rather than direct providers of case work services, and second, to establish additional services including, for example, testing of blood pressure and infant weight and height measurement for a small fee. FINAL REPORT 13 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 2.2.5 Community Habitat Foundation, CHF Community Habitat Foundation, CHF, has a number of activities funded by USAID. Those efforts primarily devoted to non-health activities are examined elsewhere in this report. With regard to health, CHF’s key focus has been its social investment initiative (SII), modeled after the World Bank and others’ "social funds." SII is funded at a $5 million level over 22 months and works with more than 180 community groups in 16 regions across Azerbaijan. These monies are used to fund grants for social infrastructure development including rehabilitation of community schools, health centers, roads, and water projects. In large part, the community groups with which CHF/SSI works are those that have already become active as a result of prior efforts by another of its AHAP partners. Thus, for example, of 152 awards (only some part of which have been for health-related projects) made under SSI by CHF, 37 were in support of Save The Children communities’ applications and 27 represent IMC referrals. CHF SII awards are capped at $20,000 per project, and community applicants are required to put up five percent of project costs in cash and fifteen percent through in-kind contributions. In considering applications submitted by communities, priority is given to rehabilitation over new construction activities. In addition to support funding for community projects, CHF under SII provides design and engineering assistance to applicants. Goals of the SII program include integration of IDPs/refugees into the broader community, and long-term community action training, to further the process of community involvement and priority setting for self-improvement. SII can be viewed, appropriately, as one of the USAID￾supported carrots available to provide incentives for communities to continue the process of "boot￾strapping" their own development. 2.2.6 International Rescue Committee (IRC) The International Rescue Committee, IRC, has been active in Azerbaijan since 1993. Its earliest efforts were devoted to improving housing for IDPs and refugees. IRC’s programmatic portfolio is very broad, including a new "cluster project" award from Mercy Corps, microloans, and agricultural improvement. The subset of IRC activities relevant to S.O. 3.1 but non-health in orientation, are described elsewhere in this report. Health related and health relevant efforts to be noted here include, first, community development and, second, following the waiving of Section 907, the creation of links between and training for communities and municipalities regarding roles, rights, responsibilities and limitations. These activities are conducted in the Southern section of the IDP belt. IRC holds, in addition to other awards, an integrated community development program (ICDP) grant from Mercy Corps; one component of the ICDP award is health. Community empowerment efforts under ICDP are identical to those use by IRC elsewhere and are straightforward. A Barda-based team enters and canvases a community. Providing training in community organizing and information about its activities through public forums and discussion, FINAL REPORT 14 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation interest is developed. That community interest leads organically to the formation of an elected community committee responsible for assessment, with the broader community, of problems, priorities, and opportunities. The potential for the community to work together with IRC to develop projects in health with seed monies and small grants is an incentive for communal action and a motivation for fund raising and in-kind work with which to realize development opportunities. Moreover, under the ICDP, IRC is able to support a multiplicity of small activities with an area: its support resources include community and economic development tools as well as health. IRC has supported through small projects activities in the development of potable water supplies, the rehabilitation of medical clinics and facilities, and the enhancement of community public health education. In addition, through a subcontract with IMC, IRC provides practical training to community level health personnel. The goals of these efforts in health are the enhancement of primary care enhanced reproductive health at the community level generally, and building sustainability in the health sector in order to enhance community development and IDP/refugee integration. 2.2.7 Save the Children Save the Children, often referred to simply as “Save,” was the first umbrella agency in the area, with responsibilities extending over the three nations of the Caucasus and overseeing primarily humanitarian services. During its umbrella period, Save, by design, had no operational programs or programmatic activities. Following Mercy Corps’ successful bid for the umbrella cooperative agreement, Save was awarded a USAID project outside the umbrella for a Participatory Microproject Program, the first such effort in Azerbaijan. Through its July 2001 award to develop broader clusters of communities, the Integrated Community Development Program (ICDP), now mirrors the set of AHAP partner responsibilities and activities. Looks toward broader geographic impact, and to build and support community empowerment and mobilization using a variety of small infrastructure projects as incentives for community and communities to work together. Eight community clusters are targeted over the grant’s 28-month life. In each of the clusters, there has been substantial prior involvement by Save and its AHAP partners. Save’s ICDP program includes activities in economic opportunity, agriculture, and business development services as well as health. As in the other community-development projects, whether in health or not, the goal of Save is to secure substantial community participation in selection of priorities and a requirement for broad community participation through both financial and in-kind contribution to project realization. Under ICDP, however, Save will seek to organize groups of communities, using inter￾community information exchange to broaden the impact geographically. Directly health-related projects have included health facility rehabilitation and development of portable water supplies. FINAL REPORT 15 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 2.3 PROGRAM EFFECTIVENESS In virtually all of its current S.O. 3.1 activities, USAID funded subgrantees under the Mercy Corps umbrella begin with process in order to achieve product. Through participatory training, communities are supported to develop new attitudes directed toward change, empowerment, and the realization that self-help can be a potent force. Individual project accomplishments, whether in health or other sectors, are the basis for a community’s enhanced morale, willingness to work together to accomplish further common goals, jointly take on new collective responsibilities, and to see themselves as actors and doers, rather than simply the recipient, for good or ill, of the actions and decisions of others. On this basis, with few exceptions, all of the USAID supported activities in health must be viewed, provisionally, as successes. "Provisionally" because too short a period has passed to judge whether the determination to have some say in their destinies will prove sustainable in the long run. Attitudes and behaviors of individuals may not subject to overnight change; those of communities, cultures, and societies may often be even more difficult to mold into new forms. In most of the sites the Evaluation Team visited, the duration of interventions has been, at most, two or three years. Yet even with such caveats, the extent to which "we can do it" attitudes have been visible is quite remarkable. Time after time, community representatives and the "man on the street" in communities we visited expressed a powerful sense of pride and accomplishment in the activities they had undertaken with, and without, external financial assistance. The Evaluation Team observed volunteer labor working on more than one project, and many had dug deep into nearly empty pockets to contribute toward the community’s share for a grant. Most spoke of "next steps" planned: working with each other or a neighboring community to prepare a water line, fix a road, repair the school or health center, and represent their remaining needs and aspirations to potential donors. Their goals were practical, not asking for a handout or the moon, but for limited assistance to help them obtain their goals. Kukand is a small village of approximately 2,000; its 365 families include 75 IDP families. CHF SII activities began in February 2001 in response to the community’s request for assistance in accomplishing its highest priority, rehabilitation of a two-room health facility. Technical review and assistance concluded that the building was beyond economical rehabilitation and would, in any event, be too small for the community’s needs. Plans were drawn for a new, six room facility, at a total cost of $25,000. The community raised a required five percent match in cash and provided $3,000 in voluntary labor. The ExCom and municipality also participated, providing donated land for the site, $100 in cash, and both graded the site and hardened the driveway, parking area and land immediately surrounding the site. In order to provide potable water, the community installed 7.5 kilometers of pipe and strong along it several taps to provide potable water to the community along its route. For the first time, potable water is proximate to the community’s residences; water was fed to all examining rooms in the facility. IMC provided training and medical “black bags” of the most basic equipment to the medical staff through IRC funding. The facility provides family planning, reproductive, maternal and child health, as well as primary care to all in the community and several surrounding villages. The population has been taught basic first aid training and preventive public health education. Commitments for the provision of examining tables and additional equipment that CHF had received have, to date, not been fulfilled. FINAL REPORT 16 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation With the objective of attitudinal and behavioral change, the effectiveness of individual health projects is not to be discounted or dismissed. Community perceptions of the quality of health care being received, and for health staffs and community users, the value of professional and public health awareness and education training received was without exception seen as substantial. This seems to be especially true with regard to maternal and child health education. Access to potable water at the community taps, made available through several Mercy Corps partners’ projects, was for all an enrichment of great value. Where pharmaceuticals were made locally accessible through revolving drug funds, the value of their accessibility without long travel time and costs was recognized and appreciated. The pride with which those responsible for overseeing collection of charges for services rendered and disbursement of funds for incurred costs wished to share with the Project Team the details of their accounts was remarkable. Further, of course, patients being served by the various grantees are simply getting care of better quality than was previously available to them. In addition, various of the S.O. 3.1 projects represent "new thinking" for the communities served and, even, broader policy discussions and formulation in future. The concepts of revolving drug funds and insurance for health care will be important when, finally, the MOH and others turn seriously to issues of further reforms in the financing of health care, academic training for physicians, primary health care, and system sustainability and reform. The Evaluation Team noted and wants again to remark upon the degree of partnering, cooperation and coordination among and between Mercy Corps’ grantees. Quite simply, no Team member has seen an equivalent or similar level of cooperation and coordination between donor-funded grantees. Against these general conclusions, two exceptions stand out: Pathfinder and AIHA. As noted above, Pathfinder is the only grantee observed whose activities, in their totality, were by intent simply service oriented. Certainly, the beneficiaries with whom the Evaluation Team met expressed substantial appreciation for those services and for those providing them. In several instances observed, caseworkers and beneficiaries seemed to have established close friendships as a result of their interactions over time. Community development, attitudinal change, and a sense of empowerment never arose as spillover benefits. In a country with so strong a history of hospital importance and underutilization concurrent with the practical absence of quality primary health care, the marginal utility of the AIHA activity in terms of needs and resources, seems at best questionable. This is not to say that science-based training of hospital physicians and specialists is irrelevant or unneeded – for surely it is not. The Evaluation Team questions whether, at this juncture, the AIHA approach used here represents the best use of finite US and USAID resources. Large parts of the AIHA award are apparently expended in travel to centers of grand technology that are achieved through, for Azerbaijan, unimaginably unaffordable costs. We suggest that a far more modest effort in the training of hospital-based physicians, using curricula and techniques that are closely aligned with and reflective of available Azerbaijani technologies and resources, might well be more practical, efficient and cost-effective. FINAL REPORT 17 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 2.4 ISSUES/CONSTRAINTS Key issues affecting all health care delivery-related projects are the future role, policy direction and funding of the Ministry of Health; continued growth of the economy; and GOAZ willingness to share the results of that growth more equitably across the population. The Ministry remains the source of approximately half of all health care expenditures in the country, but these are less than two percent of GDP or approximately $7/per capita. Until and unless this amount rises, or the current financing system with regard to both sources and amounts is very substantially modified, there are simply insufficient funds in the system as a whole to secure significant increases in the general population’s health status. Moreover, currently available MOH resources support a highly inefficient delivery system, with too much spent on hospital care and, both relatively and absolutely, too little spent on primary care, disease prevention and health promotion. To the health economist and analyst, it is remarkable – even miraculous – that the apparent improvements in health secured over the last five years, such as the increased life expectancy at birth have been obtained. The Evaluation Team has been informed by Mercy Corps staff that the MOH has determined that in future, all health training curricula will be subject to its vetting, and that approved training will in the near future be provided only by MOH staff trainers. This is fully consonant with the recommendation of the February 2002 MEDS Report, Social and Health Assessment of Residents, Refugees and Internally Displaced Persons in Azerbaijan. Per se, the Evaluation Team even agrees that, in time and with major reform in the MOH, this is a desirable outcome. But based on our observations, we vehemently disagree that that time is now or, even, in the near future. We are, instead, concerned that the general recalcitrance of the MOH, the still-Sovietized style of the medical curriculum and its presentation, and the general and apparent absence of openness to new ideas, the paucity of science-based medicine and protocols, and the continuing enthusiasm for hospital-based over primary care will lay a dead hand on practical, open, interactive and needed training if responsibility for it shifts now to the MOH. With regard to the future of USAID support of health care, two key considerations arise. First, USAID’s resources are insufficient to meet all the useful and good opportunities to assist that are available in Azerbaijan. With finite resources, choices have to be made. Second, community level health services’ projects are demonstrably a useful wedge through which to initiate broader community activities. They, like other individual projects are seldom, by themselves, enough to secure the broader attitudinal, behavioral and economic spillover necessary to community takeoff. The Evaluation Team, like the several partners, Mercy Corps and USAID see the incentives posed by the possibility of further financial assistance to follow-on projects as important. Hence the value of multiple partners and programs in the field, and of the cluster approach to spread gains across a broader area. On the other hand, at the humanitarian level, these projects bring individuals and communities a real and immediate benefit: improved health. If, then, resources are narrowly targeted to a small number of communities, synergies exist which may move those communities rapidly forward – however, with regard to basic humanitarian needs, many will be left behind. Alternatively, spreading limited resources broadly across a larger number of communities, while providing benefits to many, may well result in critical mass and sustainability being accomplished nowhere. FINAL REPORT 18 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation As always, the determination of spread versus concentration lies with USAID. But the Evaluation Team concludes that relatively narrow targeting is a necessary evil given resource constraints and the scope of needs in Azerbaijan. 2.5 SUSTAINABILITY/INTEGRATION/TRANSITION As addressed above, the judgment of the Evaluation Team regarding sustainability of process, attitudinal and behavioral changes must be provisional. So, too, must be the conclusion regarding financial sustainability of a number of the health-related projects examined. Revolving drug funds found seemed to be fully successful, both in Naxcivan and mainland Azerbaijan. The privately operated Fizuli hospital in Babi, built with support to the community by CHF/SII and IMC, is clearly a financial success and its sustainability is assured. The future of other projects, however, is less clear. For example, one of the SII projects visited was a newly built primary healthcare facility, replacing one no longer worth rehabilitating. The building is attractive but virtually empty of the modest equipment needed to render primary care. Assurances the Mercy Corps grantee, CHF, had received prior to construction that necessary equipment would be supplied by the MOH have not yet been met although the building has now been completed for some months. It should be noted, however, that the AHAP partners have learned from this experience. IMC has developed a protocol detailing supplies and equipment necessary to the successful operation of such a facility and this protocol will become a part of future projects. As part of the funding of such projects in future, grant funding will be included to provide necessary operational supplies and equipment, as well as for construction of the facility itself. Another example: a birthing center visited in Naxcivan (see the Naxcivan section) and connected to ADRA has in place a revenue-sharing arrangement with the district hospital. This income should be more than sufficient to assure the birthing center’s financial sustainability. But the revenues expected have been used by the hospital and not returned to the community because the hospital itself is inadequately funded. Yet another: the sustainability continuation proposal of Pathfinder, to provide first aid (basic wound management), blood pressure monitoring, measuring of child weight and height, and basic drug provision (aspirin, antibiotics) for a fee, seems to both Pathfinder’ staff and to the Evaluation Team totally inadequate to continue the service levels now provided once USAID/Mercy Corps financial support is terminated. Several of the partners including IMC, CHF, ADRA offer some form of health insurance, usually on a family subscription basis, each with a benefit package that may vary; several of these also offer fee-for-service care and IRC offers loans for essential health services beyond the immediate capacity of patients to pay. Take-up rates are highly variable across partners and sites. Health insurance is a new concept to all the communities and the expansion of program possibilities that flows from exposure to new ideas is highly desirable. But whether or not “premiums” can be set high enough to cover all relevant costs and sufficient participation rates can be maintained is not yet clear. FINAL REPORT 19 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Again, however, the Evaluation Team notes that financial sustainability of many of these projects and their approaches to cost recovery may ultimately depend upon decisions by and about the Ministry of Health. Will private practice be allowed to blossom outside Baku? What will be the role of district head doctors and other officials in sanctioning benefits and cost recovery approaches? Will the MOH work with others and pay its “fair share”? What level of financial support will the MOH receive from government income? None of this should be taken as pessimism on the part of the Evaluation Team. Rather, we recognize that remarkable change opportunities are at hand and a broad array of alternative sustainability approaches are being tested. The outcomes, however, are not yet clear. Nonetheless, the core of the argument for sustainable health projects was well put by IMC’s cost recovery presentation at the October 8-9, 2002 First National Conference on Community Development Best Practices and Future Directions. A part of that presentation: Cost recovery around health is an excellent vehicle for community development. Cost recovery in health is a very complex phenomenon. It has capacity to mobilize resources to address essential health needs in communities. Mobilization of resources requires skilled local management Skilled management requires robust democratic process. ` Democratic mobilization leads to enhanced civic participation among all community members and strengthens civil society. Financial resource mobilization, democratic governance, and transparent accounting practices lead to economic development and sustainable community based health care. In the context of S.O. 3.1, sustainability in the arena of health will require the dedication of resources in excess of those currently made available by donors. The active engagement of the Ministry of Health will be necessary to sustainability. Similarly, the engagement of the formal institutions of Municipal and Ex Com authorities can support the community development of villages that have not been directly impacted by S.O. 3.1 programming. 2.6 POLICY AND RELATED ISSUES We conclude that health is an appropriate part of S.O. 3.1 and recommend that it should appropriately remain there. The projects observed demonstrate that health can be an appropriate fulcrum on which to lever broader community activities and development. It is less clear, however, that its focus should be necessarily limited to the IDP belt since needs for improved health care appear to occur clear across much of the country. We suggest to USAID that there should be a conscious decision to concentrate health and other S.O. 3.1 funds in a limited number of communities to develop over time a critical mass of support for attitude and behavior changes necessary to achieving community and economic development. Certainly the multiplicity of difficulties regarding USAID working with the Ministry of Health is recognized. Nonetheless the conclusion cannot be escaped that in health, sustainability, indeed, even viability, is dependent upon MOH decisions and that those decisions at least may be better if FINAL REPORT 20 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation outside assistance can be brought to bear. Moreover, in their day-to-day activities, the INGO partners of Mercy Corps not only must deal with ExCom officials, municipality leaders, but, too, district hospital head doctors and Ministry of Health officials. While one respondent claimed this to be easy or quick, most said that with negotiation and persistence, accommodation had been reached. Three events will soon occur, each of which may provide USAID the opportunity to deal from a relatively protected position with the Ministry of Health. A contract will soon be let, on behalf of the Ministry of Economics, to study health care expenditures in Azerbaijan. USAID has supported similar work in a number of NIS countries. USAID could indicate its interest in at least observing, and perhaps at a very modest level, participating in such a study. A possible point of focus, perhaps, is the considerable volume of health activities supported by USAID in both the IDP belt and Naxcivan. The second event is the expected release of the CDC reproductive health study very late this year. The interim report enjoys the imprimatur of both USAID and the MOH. The final report is expected to show maternal and infant morbidity and mortality outcomes far worse than those of the MOH. If resources are available, “packaging” the CDC report with an offer of modest assistance to help address these problems should be considered. FINAL REPORT 21 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 3 ECONOMIC OPPORTUNITY PROGRAMMING The Partner organizations under a total of seven subgrants are engaged or have engaged in economic opportunity programming under the AHAP umbrella. ACDI/VOCA conducted a program of Community Economic Recovery Groups (CERGs) that has been extended, removed from the AHAP umbrella, and is placed into S.O. 1.3, Economic where it is managed directly by USAID/Azerbaijan personnel. 3.1 SUMMARY OF OVERALL PARTNER PROGRAMS Over the past two years, the Mercy Corps umbrella managed a program of, on the average, about 14 subgrants to eight Partner INGOs. These subgrants were for programs in the four USAID conflict-affected areas of Central and South IDP Belt, the Urban area near Bapu, and Naxcivan. The Table presented below shows a summary of the engagement of the Partners in economic opportunity programming as well as in health and community development and indicates the regions of activity. Table 3-1 Summary of AHAP Subgrants by Conflict-Affected Region No. of Central South Urban Naxcivan Grants ACDI/VOCA 1 CERG (EO) ADRA 2 Microfin. (EO), Health CHF 2 SII (EO) CEEOP (EO), SII (EO) SII (EO) IMC 1 Health IRC 3 Health CD, ICDP (EO) Save (SC) 3 ICDP (EO), CBLS, (EO) CD Pathfinder 1 Health World Vision 1 CD Totals 14 6 5 2 2 3.2 ECONOMIC PROGRAM REVIEW In the Table above, CHF SII (Social Investment Initiative) is shown in three of the four conflict areas and was implemented under a single subgrant. This is the sole subgrant to be conducted in more than one USAID impacted area. Save the Children and CHF are the Partner organizations with two separate subgrants for economic opportunity. ACDI/VOCA conducted a program of FINAL REPORT 22 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Community Economic Recovery Groups (CERGs) that has been extended and removed from the AHAP umbrella. ACDI/VOCA targeted and realized the creation of a total of eight Community Economic Recovery Groups (CERGs) under an AHAP subgrant. These CERGs were primarily farmer associations at the regional level. Associations were linked with other suppliers and customers through trade fairs (marketing chains). A major point of leverage was investment grants of up to $15,000. The Evaluation Team visited food processing, oil seed, and wool cleaning projects that were developed by entrepreneur groups under investment grants. ACDI/VOCA provided technical assistance for these projects through feasibility studies and market surveys. The Adventist Development and Relief Agency (ADRA) conducts a microfinance project in the Naxcivan Autonomous Republic. Loans are made to individuals and to groups. These loans are to agriculturally oriented activities and to manufacturing and trade entrepreneurs. In excess of 7,000 loans have been made to date and there are 2,700 current clients with loans; many of these clients are on their fourth and fifth loans. Group loans typically have about ten member clients. Repayment rates have been excellent and are at approximately 98%. The Community Habitat Foundation (CHF) has a subgrant for a Social Investment Initiative (SII) that supports the funding of larger scale community projects up to about $25,000. SII can include economic opportunity efforts such as irrigation projects and has primarily supported school and health center rehabilitation. The Community Employment and Economic Opportunity Program (CEEOP) operates only in the South region of the IDP belt and CHF is serving as the facilitator for this effort under an AHAP project extension. CEEOP is a program, similar to agricultural extension services, that is designed to increase the capacity of local producers through business and management training. Under CEEOP, CHF has conducted marketing studies and training in cooperation with IRC community mobilization efforts. A cost recovery and sustainability exercise, CEEOP training has generated revenues of $1,600. The International Rescue Committee (IRC) is engaged in economic opportunity programming through its Integrated Community Development Program (ICDP) in the South region of the IDP belt. An emphasis of the extension programming is two Regional Resource Centers that will provide business development services. Under the ICDP effort, IRC has provided small grants for food processing plants. The Evaluation Team visited a cheese factory and food preservation projects that were developed by IRC and were in full operation with plans for plant expansion. Save the Children (SC) has been involved in economic opportunity through its ICDP and Community Based Lending and Saving (CBLS) subgrants that operate in the Central region of the IDP belt. The SC ICDP is oriented to sustainability through the use of business development services by the local NGOs. Kosia Smeda, a local NGO, provides 25 trained extension specialists to work with community groups and has targeted the development of 7 Agricultural Information Centers (AIM) to provide agricultural extension support and business development services (BDS). Similarly ARAN, another local NGO, provides trained paralegals to provide assistance to community groups on social and legislative matters as well as on government relations and pension issues. The CBLS program provides loans to individuals and to lending groups, basically trade-oriented groups. FINAL REPORT 23 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The three remaining AHAP Partners, International Medical Corps (IMC), Pathfinder, and World Vision are not directly involved in economic opportunity programming. The IMC is conducting cost recovery components for health services which do have economic ramifications. 3.3 FINDINGS AND CONCLUSIONS The EO sites visited by the Evaluation Team showed an impressive level of success. A contrast of impact was seen at the wool-cleaning project when the ditch that had previously been used was pointed out. It was contended that working in that ditch to clean wool resulted in illness and constituted a hardship to achieve wool-cleaning results. The Evaluation Team observed women in the Belasuvar Camps who were using ditches for washing raw wool as well as for washing clothing. Loan projects conducted by SC in the Central region of the IDP belt and by ADRA in Naxcivan appeared to have supported borrower enterprises with relatively few failures. From interviews with loan groups, the Evaluation Team estimated that one in ten loans did not accomplish the borrowers economic objectives. Repayment rates were at approximately 98%. The conclusion from Evaluation Team visits to these Partner selected sites is that the loan projects were working as planned. Three focus groups, composed of persons with no or very little contact with AHAP programming, were conducted. Two were in the Central region and one was in Naxcivan. Individuals in these groups, in addition to identifying other community needs, stressed the lack of economic activity in their areas. The communities visited in the urban area near Baku were noticeably in need of additional economic activity. The obvious conclusion from these focus group sessions and by observation in the urban target area is that economic opportunity projects are widely needed. 3.4 RECOMMENDATIONS The economic opportunity projects conducted under SII supported community groups in designing and implementing projects of community wide interest and impact. These projects were in irrigation and agricultural product processing. Removing these kinds of economic opportunity projects from S.O. 3.1 would weaken the community mobilization and community development efforts that have been shown to have an enhanced effect through addressing economic opportunity concerns. The microfinance projects of Save the Children and ADRA, on the other hand, have not demonstrated that they contribute significant synergism in support of either community development or health programming. Such projects and grants could be moved to direct USAID management under S.O. 1.3, Economic Development. In making that move, additional regional responsibility will result for the USAID/Azerbaijan personnel who manage S.O. 1.3 since these microfinance projects are currently in Naxcivan and the Central Region of the IDP Belt. FINAL REPORT 24 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 4 COMMUNITY DEVELOPMENT The term “community development “can embrace a wide range of concepts and interpretations. Within the AHAP context Community Development includes the initiating of the learning process of a community, community organization, the mobilization or activation of community energies and resources, and the definition of community goals and the placing of priorities on possible projects. The next steps of actual project implementation are also included in Community Development under AHAP programming. In a broad interpretation, Community Development can include programs and projects of all types from basic human needs of food and shelter to projects of economic opportunity. Projects to benefit individuals and select groups such as loan programs and business development services are classified as Economic Opportunity programming for the purposes of this report, although the development of the community can be affected by these efforts. 4.1 COMMUNITY MOBILIZATION AND DEVELOPMENT Community mobilization or community activation, which is the organization of community members and formulation of community direction is conducted by all of the subgrantees to some degree. Community mobilization has been expressed by Mercy Corps personnel to be the process of individuals coming together as a group, discussing community issues, and deciding what to do. Participatory Rural/Rapid Appraisal procedures, according to Mercy Corps personnel, are used by the subgrantees. Awareness of environmental impact of community projects is also an element of all of the subgrantee programs. The table below presents summary information on the Community Development efforts of the AHAP subgrantees. 4.2 FINDINGS AND CONCLUSIONS The Evaluation Team found that all of the subgrantee programs affected the communities in which they operated in one or more ways. ACDI/VOCA created eight Community Economic Recovery Groups (CERGs) which were primarily community farmer associations. ADRA, operating in Naxcivan, created 46 village health councils and other community groups around their loan programs. CHF funded larger scale community projects which impacted on the confidence and self-satisfaction of the participating community groups. Save the Children leaves behind skilled groups with the ability to write proposals to support their realization of other donor-funded projects for their communities. IMC trains health cost recovery groups and health practitioners that impact on the health and well-being of community members. A community health management committee (CHMC) is responsible for building consensus around identification, prioritization and proposed solutions to the health problems it faces. Pathfinder, through its health specialists program, works to improve the health of targeted community members. World Vision, in the Sumgait area close to Baku, applies an urban model of community mobilization that is similar to that of IRC. The Evaluation Team visit to the World Vision site in Sumgait was to an IDP settlement that, except for the lack of access to agricultural pursuits, resembled the several IRC sites that were visited in the South region of the IDP belt. FINAL REPORT 25 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Table 4-1 Summary of Community Development Activities of AHAP Subgrantees Mobilization Project Additional Information and Basic Methods Funding/Support ACDI/VOCA Market Chains Provides equipment Environmental Ongoing TA, follow-up and materials Training ADRA Establish village None for projects Focus on loans and health, councils, TA 46 village councils formed CHF Integrate with other Partner SII: $20,000, Community share of Training, linkages to EO programming Community groups 20% IMC Health groups Medical equipment Health cost recovery TA and training IRC PRA emphasis Micro: $3-5,000 Partner linkages, Initial 4 day training secondary funding Save (SC) Training needs Micro: $3-5,000 Integrate with local assessment, PRA NGOs & SC EO program Pathfinder Train health None Health workers reach workers 1,000 residents World Vision Urban PRA Micro: $3-6,000 Project funding starts Phased with small ($100) projects Of the Partner organizations, IRC appeared to have the strongest and most coherent program of community mobilization. IRC utilizes a proven participatory training process that has been developed and applied in a variety of other IRC country settings. The IRC community mobilization begins with a four-day intensive program for up to 150 members of a community or collection of communities. Subsequent training is provided and support is given for microprojects that in subsequent cycles can be as large as $5,000. Several of these community groups were supported in realizing their community priorities through linkages with CHF. The CHF program provided grants in the $20,000 range that allowed these communities to develop substantive and quality community projects. 4.3 RECOMMENDATIONS The Evaluation Team recommends the inclusion of economic opportunity as important to Community Development in those instances where the majority of the community population directly benefits. An example would be an irrigation project where the community consists primarily of farmers who are all directly affected by such a project. Accordingly, in current and future Community Development programming, these cooperative-type community efforts should be permitted under the technical assistance and funding that is provided by the Partner organizations that are engaged in Community Development. FINAL REPORT 26 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Larger funding for community projects available through CHF was observed to have resulted in quality efforts, an increase in community pride, and a strong sense of community achievement. The Evaluation Team recommends that future community programming be structured to allow community project funding in the $15,000 to $25,000 range. Where gradual phasing of the amounts made available to community groups is used as a training mechanism, the smallest grants should still be sufficient to impact on the typical projects that have historically been undertaken in that region. The Evaluation Team recommends a minimum of $5,000 (large enough to assure that in a community structure such as a school that the roof will provide a sufficient cover from the elements). Other Partner organization project efforts in Azerbaijan could benefit from the application of the community mobilization/ development methodology of IRC. While participatory methodologies are applied to one degree or another, the structured approach of IRC was seen to result in strong community groups that accepted the responsibility for planning and taking actions to improve their communities with or without international donor support. Achievement of such a level of self-actualization/realization does not occur without a substantial commitment to support such a community breakthrough. The Evaluation Team recommends that current and future community development efforts require Partner staff training in participating methodologies to the extent that similar support can be provided to other community groups in the implementation of the various AHAP approaches to community development, health, and economic opportunity projects. Khoruzuli Clinic is located in a small village close to another village in which IRC had been undertaking community training and empowerment activities. As word of this program spread, the Khoruzuli community approached IRC to request assistance. With rehabilitation and training, the health center and its medical staff are now providing, primary and well-baby care to a population of 2,165, of which 131 are IDPs in three villages. The patient load is estimated at 300-350 per month. Immunization services are provided at the clinic on the 28th day of each month, accompanied by active outreach and follow-up efforts. Painted on the front of the facility as a constant reminder is a table of appropriate immunizations and the age at which each should be preformed. Potable water is available in the building’s courtyard, but not inside. The project has trained trainers of trainers in disease prevention and health promotion. As part of sustainability efforts, just beginning, the clinic is providing diabetes detection, pregnancy and other testing services for a small charge. A community committee and a facility sustainability fund committee are in place and active. On its own, this active community has undertaken street cleaning and tree planting as well as small scale environmental efforts. ACDI/VOCA is working with the community to supply it with improved oil sunflower seed. The community has submitted to CHF an application to assist it in improving the electrical supply. FINAL REPORT 27 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 5 NAXCIVAN AUTONOMOUS REPUBLIC The Naxcivan Autonomous Republic lies to the west of mainland Azerbaijan, and is physically separated from it by the southern extension of Armenia. Naxcivan, bounded by mountainous borders with Turkey, Iran and Armenia, is estimated to have a population of approximately 358,000, of whom some 91,000 are children under the age of 12. Seventy thousand live in the capitol and major city of the same name. There are 270 villages but of the Naxcivan Autonomous Republic’s population, only some 4,000 to 6,000 or about one percent of the population are IDPs and refugees. Economic conditions, community development, and the nature of USAID-supported efforts in Naxcivan appear to be approximately two years behind those found by the Evaluation Team elsewhere in Azerbaijan. This reflects the underlying realities of the Naxcivan Autonomous Republic, home of the Azerbaijan president and several key members of his government. Naxcivan is afflicted with all the problems found elsewhere in Azerbaijan: community infrastructure of all sorts in disrepair, loss of traditional markets and trading patterns, and an education sector inadequate to the Republic’s needs. In particular, the government health care system has inadequate resources. Care is focused in massively underutilized hospitals rather than primary care and there is an overabundance of specialty-trained physicians. Along with the traditional problems of infectious and, increasingly, chronic diseases which afflict the population, goiters have of late become an increasing problem, especially in a number of the mountainous villages. To address this problem, the salt mill in Naxican was provided iodizing equipment by the UN. But because of some combination of equipment maintenance and cost, the Evaluation Team was informed that the mill iodized its product for only a brief period. In Naxcivan, problems have been worsened by two additional constraints. First, it is effectively isolated economically and geographically from the rest of Azerbaijan by Armenian territory stripped from the former borders of the country by the Soviet Union. Prior to the war between Azerbaijan and Armenia, Naxcivan was connected to the "mainland" by a railroad link across Armenia that made commerce between the two parts of the country economically viable. With the war, Armenia broke the rail link between Naxcivan and mainland Azerbaijan. Now, only very high value-added products justify the expense of air shipment or the tortuous and problematic land shipment through Iran and Turkey. The chief products of Naxcivan are agricultural and few have a value added component that meets this case threshold. Second, as a result both of the war and its subsequent isolation, Naxcivan in a number of ways is simply at a lower level of redevelopment and growth than the mainland and its needs are in many ways more immediate and severe. The entirety of the Autonomous Republic and its population are considered to be, and are, conflict-affected. FINAL REPORT 28 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation As one measure of the effects of war and isolation, the only Mercy Corps-supported NGO grantee, ADRA, did not terminate its feeding program until mid-2001 and the need for direct care emergency services has until very recently driven virtually all of its health activities. Even in its receipt of external assistance, Naxcivan is isolated. In addition to ADRA, there is only one other international NGO, the International Organization on Migration (IOM), that operates there. IOM also provides humanitarian assistance. 5.1 PROGRAM/PROJECT DESCRIPTION ADRA has been involved in Naxcivan since 1993, originally as a U.S. Department of Agriculture direct provider of food to the Naxcivan Autonomous Republic’s population following the war between Azerbaijan and Armenia. Since then, its agricultural activities have expanded to include provision of improved seed, agronomy training and assistance to farmers, and reforestation efforts – the last necessitated by the denuding of forests after 1991 to provide fuel as a result of Armenia’s breaking of the land access and rail link to the rest of Azerbaijan. In health, beginning in November 1997 and currently funded through a grant from Mercy Corps, ADRA’s chief efforts were and continue to be devoted to direct primary health service provision using nurse-staffed fixed health points supported by mobile physician health units – a gynecologist, pediatrician and senior nurse – who visit the health points on a periodic basis. The majority of population – as much as 70% of the 270 villages – has access to ADRA health points. These provide health education, disease prevention and health promotion with a focus on maternal and child health, primary care and limited emergency services. They also provide a limited formulary of drugs on a cost-plus basis through a fully self-supporting revolving drug fund initiated by ADRA. In 1997, ADRA was the key provider of health education and services that successfully controlled a major outbreak of diphtheria in several areas of Naxcivan. This success, and the general perception among members of the communities with which the Evaluation Team spoke, has been key to ADRA’s acceptance and the winning of strong support from both the Naxcivan Ministry of Health (NMOH) and the communities themselves. At their maximum, ADRA supported 46 but has since reduced the number of community-based health points to 44. Further reductions, to approximately 30, are planned. Access is being maintained, however, as the remaining points are being further strengthened to serve larger clusters of communities and as the NMOH takes on additional responsibilities. In the absence of capacity by the NMOH, from 1998 until August 2002, childhood immunization services were also provided by ADRA. Since then, it has removed itself from immunization except that, in significant areas of Naxcivan, ADRA is in charge of the cold chain, moving NMOH product to refrigerators – often with associated generators to protect refrigeration – and supplying it to NMOH physicians who deliver it to patients. ADRA’s intention is, ultimately, to turn over cold chain responsibility to the Ministry and anticipates doing so in the foreseeable future. ADRA is in the process of slowly reducing its direct provision of health care as the government’s capabilities increase. Over the last year, since the Presidential waiver of the Section 907 provision, FINAL REPORT 29 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation direct activities have been undertaken by ADRA to strengthen the Naxcivan Ministry of Health (NMOH). Work with the Republic’s NMOH has focused on (1) training and (2) maintaining and improving the vaccine cold chain. Training has primarily involved inclusion of NMOH physicians and nurses to improve the quality of care provided in ADRA’s health points. This training is usually provided through day-long sessions, using ADRA staff to present a practical curriculum reflective of the needs and realities found in the field, and focuses on primary care. The Evaluation Team had the opportunity to talk with the head doctor of the raion (regional) hospital in Mamukend, in which ADRA/NMOH joint EPI training of six nurses was observed. He was strongly supportive of the ADRA effort and expressed pride on behalf of his hospital in serving as the site for the training. In conversation with nurses at the birthing center – in which 175 births had occurred in less than two years – and the one health point visited, we heard substantial praise for the training ADRA had supplied. It was described as providing new, useful, and operationally practical assistance that was directly on point to their needs. While continuing to provide direct medical services over the past two years, ADRA has shifted its activities to an increased emphasis toward community development. In 1999, Village Health Councils (VHCs) were introduced, using health as the basis for developing broader community activation. VHCs, as in community development models used by other international non￾government organizations (INGOs), are the product of ADRA-initiated activities to bring together community members to determine and prioritize among needs and to bring community labor and financial resources into being to realize them. The evaluation Team met with VHC leadership and community members in Aza, Arafsa, and Ashagi Yayji. In the three communities, community wishes had been prioritized through broad meetings and community surveys. VHC leadership had been elected by popular vote, and seemed to have been quite active in the collection of local funds for and implementation of a variety of specific community improvement projects. Limited additional support to help finance projects had been sought and received from mayors and ExComs but the overwhelming source of support had been from voluntary community labor and purses. Even before the waiver of Section 907, ADRA accepted all members of committees who were elected to serve by the community. Thus, mayors, ExCom, and other "government" members served on various ADRA community-level committees. This reflects the role of such individuals as leadership figures, as perceived by the local community. Concomitant with this shift in emphasis from direct services to community development, ADRA also sought, in 1999, a grant from Mercy Corps to put in place in Naxcivan an integrated community development program (ICDP). Because ADRA is the only Mercy Corp sub-grantee in Naxcivan, communities in the area are unable to take advantage of the collaborative IRC/CEEOP and CHF/SII infrastructure support activities found in the IDP/refugee belt of the mainland. Through the proposed ICDP, ADRA would have had available the resources to support small FINAL REPORT 30 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation grants to communities, for their use in helping to realize accomplishment of some of their priorities. ADRA did not receive the Mercy Corp ICDP grant it had sought. Instead, it was urged to seek some of the benefits of integration found elsewhere by linking together its activities in health with its microfinance efforts. To some extent it has been successful in accomplishing this by using its health points as a source of cross-reference to the availability of microfinance loans. And, indeed, there have been a few cases in which this has occurred. On the other hand, the relationship between these two activities in the support of community development is, at best, tenuous. Community development has been an ADRA focus only for the last two years and there are no other Mercy Corps Partners in the Naxcivan Autonomous Republic to help support community development. The funding of the infrastructure through cluster and other project activities has been key to the community development observed by the Evaluation Team elsewhere in Azerbaijan. It is not surprising that in Naxcivan one does not find the kinds of externally funded community development projects observed throughout the IDP belt in Azerbaijan. What is surprising, rather, is the extent to which the Evaluation Team in its limited observation found that the community development process accomplished through VHC support has bolstered broader community activities. With local funds, occasionally supplemented by financial and in-kind assistance from the ExCom and/or municipal authorities, in several places a mosque had been built, roads improved, plans for potable water supply developed, a birthing center rehabilitated and similar activities undertaken. Another VHC, visited by the Evaluation Team, had concentrated only on the health clinic which, however, had greatly upgraded the health services available to that community. No other community projects had been undertaken in that village. Recently, ADRA has introduced the concept of insurance at the community level through installation of a program of prepaid fee-for-service coverage directed at reproductive health. ADRA had, based on the frequency of inquiries, anticipated that this would be especially attractive if it were to include birth control services. After some months of discussions and negotiations with the NMOH, ADRA won approval to offer contraceptives as a core component but was directed that it could not publicly advertise this benefit. ADRA’s intention was to provide oral contraceptives linked to appropriate gynecological exam and follow up. Oral contraceptives would be made available to participants for 2,000 manats ($.40) per month and, the Evaluation Team was informed, at gynecological exams scheduled with medical appropriateness, the charge would be 5,000 manats, inclusive of that month’s supply. The prepaid fee-for-service reproductive health-focused insurance, available at all health points, was offered at the beginning of 2002. Enrollment was very low, and only some 475 women participated. At this point, ADRA staff’s tentative conclusion is that it was simply misled by the volume of earlier inquiries regarding community interest in the service, and is considering whether and how to modify the benefit package to enhance its attractiveness to increase participation. 5.2 PROGRAM EFFECTIVENESS The health program of ADRA has had substantial impact on the quality of and community access to health care in Naxcivan. As noted, community support is strong. The government, as FINAL REPORT 31 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation represented by the Speaker of Parliament, is very supportive; and the training provided health point and NMOH medical staff appears to be very practical and directly applicable to their professional responsibilities and development. The revolving drug fund has significantly increased availability of formulary drugs at the community level with the result that effective costs to the population, including those associated with travel to NMOH dispensing facilities and private pharmacies, have been significantly reduced. ADRA public health education activities through the health points has included information on diet, immunization schedules for infants and children including referrals to NMOH vaccination services, and both prenatal and neonatal care. On a limited basis, ADRA has dispensed appropriate supplements and drugs to help address the problem of goiters, and has included preventive education among its health outreach activities. Although to date prepaid fee-for-service insurance has had very limited enrollment at the community level, it represents the first introduction of insurance concepts to the NMOH. One of the five raion hospitals to be funded by the World Bank health reform project is located in Naxcivan. ADRA staff are hopeful that financing mechanisms anticipated to be tested there will form the basis for a joint presentation to the NMOH on health financing policy, and that ADRA and the World Bank project will be able to work together effectively to advance an agenda of health reform policy in the Autonomous Republic through policy education of the NMOH. 5.3 ISSUES/CONSTRAINTS It is worth pointing again to economic and geographic isolation, the absence of inexpensive transportation to the mainland, the blockade by Armenia, and the consequent disruption of external markets for Naxcivan’s agricultural products as key problems that collectively help explain the comparative backwardness of community and economic development in the area. While we have no documented economic or other data with which to support this conclusion, the Evaluation Team’s professional judgment is that conditions in Naxcivan lag those of the mainland by two years or more. A number of Naxcivan’s difficulties are externalities not readily subject to USAID’s intervention. The view of the Evaluation Team is that the most serious constraint on USAID securing its goal of more substantial community development in Naxcivan is the absence of resources with which to support relatively small grants to communities to assist them in undertaking community developed priorities. ADRA works virtually alone as the only USAID/Mercy Corps-funded INGO in the area. Although its relations with the communities and with the government of Naxcivan are very positive, its budget does not include funding to help communities move from prioritization of needs to realization of projects. As a consequence, ADRA has maintained an activity portfolio which is limited to issues of health and some microfinance efforts. 5.4 SUSTAINABILITY/INTEGRATION/TRANSITION Within the constraints of program direction and resources, on the one hand, and external conditions on the other, the Project Team has been impressed with the successes that ADRA has secured. The revolving drug fund is now and appears for the future to be fully sustainable. FINAL REPORT 32 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Consolidation and reduction in the number of health points seems to be planned on a rational basis, and is being introduced in a fashion that seems to assure that access will continue to be maintained. Although the natural linkage between a health program still fundamentally tied to service delivery and a microfinance program consisting primarily of small loans to farmers is at best weak, ADRA has attempted to use the first as an advertisement point for the second. Given the distribution of health points and the use made of them by communities’ populations, this appears to have been modestly useful. One important issue regarding sustainability is the lack of resources, at all levels, found throughout the region. While the Evaluation Team has insufficient basis for disagreeing with ADRA staff that the failure to date of prepaid fee-for-service reproductive health insurance is primarily a reflection of an insufficiently attractive benefit package, we are forced at least to ask whether and to what extent a lack of family income may also be a part of the problem. At the level of Naxcivan’s government, inadequate financial capacity is an obvious problem. But that incapacity seems also to have current and potential consequences for sustainability. One example observed during our brief site visits: in Aza, the site of the birthing clinic, a fee of 40,000 manats is imposed for each delivery by the regional hospital. Those funds are collected by the birthing center but then, in their entirety, are sent on to the hospital. Half of those funds is supposed to be returned to the community for its use. The funds which the hospital is supposed to return to the community represent the basis for the birthing center’s future sustainability, repair, maintenance, and improvement, as well as a potential resource to the community for other activities. To date, however, less than 400,000 of the seven million owed to the community by the hospital have actually been transferred back. The unreturned difference, the community, ADRA staff and the Evaluation Team are all convinced, has been used by the hospital to meet its own real costs in the absence of adequate NMOH funding. Finally, on the issue of financial resources for health, the Evaluation Team asserts that true sustainability – including the capacity of ADRA to turn over its health points to the NMOH – is fully dependent on the NMOH having the resources as well as the commitment to accept them. ADRA is hopeful that NMOH commitment will grow as assisted by the NMOH policy education which began in earnest with consideration of the ADRA proposal to offer its prepaid fee-for￾service insurance and should continue with the World Bank-funded health reform project in Naxcivan. The Evaluation Team, as well as ADRA personnel, is less sure about the availability of financial resources becoming available to NMOH in the foreseeable future. With regard to broader community development, the Evaluation Team would recommend that additional resources, including the introduction into Naxcivan of additional INGO partners, be committed to achieve this goal. The entire population of Naxcivan is conflict-affected and it is one-third the size of the country’s IDP/refugee population. The USAID/Mercy Corp. support for activities in Naxcivan has been, disproportionately more limited. FINAL REPORT 33 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 5.5 POLICY AND RELATED ISSUES USAID/Azerbaijan will ultimately determine what level of resources can be made available in Naxcivan. The USAID investment in health and related community development has been effective and efficient as well as quite limited. It has, through ADRA, produced a very considerable "bang for the buck." As argued elsewhere, the Evaluation Team sees the current state of development as roughly two years behind that elsewhere in Azerbaijan. And, indeed, the nature of USAID/Mercy Corps/ADRA activities reflect those we understand to have obtained in the IDP/refugee belt in late 1999. The Evaluation Team recommends that, if resources allow, Naxcivan should at the earliest possible date receive a further influx of USAID-supported activities similar to those made available in the IDP/refugee belt two years ago. These include enhanced INGO partnering, support for broader community development infrastructure priorities through CEEOP and SSI-like funding with special attention to potable water and cluster development. FINAL REPORT 34 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 6 REVIEW OF AHAP MECHANISM The Azerbaijan Humanitarian Assistance Program (AHAP) umbrella grant mechanism has been managed by Mercy Corps International since January 1998. It was preceded by a similar umbrella mechanism, awarded in 1993 to Save the Children/US. Both mechanisms were devised as cost￾effective ways of implementing the US assistance strategy in Azerbaijan, where the USAID presence is provided by an office in Baku, first of USAID/Caucasus and now USAID/Georgia. 6.1 PROGRAM OVERVIEW Grants under the earlier umbrella provided direct humanitarian assistance to some 800,000 Azeri IDPs and refugees resulting from the conflict in Nagorno-Karabakh. The charge of the Mercy Corps umbrella began a process of transition from direct humanitarian assistance to more sustainable development activities and was expanded to include “other vulnerable persons” in Azerbaijan. Activities were to include “a broad array of project/skill areas with a perspective towards improving employment opportunities.”7 6.1.1 Cooperative Agreement The first Mercy Corps umbrella, now called AHAP I, was let through January 2001 and extended through 2003. A further no-cost extension through January 2004 was granted later for a total value of $45,000,000. AHAP funds now are fully obligated. According to its Cooperative Agreement with USAID, Mercy Corps is charged with: • Establishing a system of sub-grants, including development and issuance of RFAs, review of applications and negotiation and award of subgrants; • Responding to appropriate unsolicited proposals; • Coordinating geographic and substantive focus; • Developing a simple system to track basic needs of target populations; • Analyzing needs and trends and developing modified programs to address them; • Developing and implementing an effective monitoring program; • Specifying anticipated results and impact; • Promoting communication and dialogue with subgrantees and disseminating information; • Helping grantees comply with USAID policies, procedures and regulations 6.1.2 Subgrants Since 1998, the AHAP umbrella has issued nine RFAs, three in AHAP I for proposals in the three main sectors and five in AHAP for a more complex set of activities. Altogether, more than 30 grants to some 14 organizations have been made. (See Annex E for a list of RFAs and grantees). 7 USAID Cooperative Agreement with Mercy Corps dated 13 February 1998 FINAL REPORT 35 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The total value of grants awarded as of 14 October 2002 is $36,079,033. (Table 6-1 for a full list of grants). Unsolicited proposals have not played a large part in the AHAP grant portfolio. Table 6-1 AHAP Subgrants as of October 2002 Subgrant # Implementing Partners Subgrant Period Approved Budget H-01 ADRA 5/1/98-4/30/00 981,252 H-02 IRC 5/1/98-12/31/99 353,700 H-03 RI 5/1/98-5/31/00 1,223,366 H-04 UMCOR 5/1/98-5/31/01 966,044 H-05 UMCOR 7/28/98-10/31/99 408,432 E-01 ACDI-VOCA 9/1/98-12/31/99 221,986 E-04 WV 5/1/98 - 2/29/00 500,000 E-03 CAD 5/1/98 - 12/31/99 252,882 E-02 AmRC 5/1/98-11/30/98 165,209 E-05 ADRA 2/1/99 - 10/31/00 578,209 S-03 WV 5/1/98 - 2/29/00 1,787,922 S-02 IRC 5/1/98 - 10/31/99 898,705 S-01 CARE 5/1/98 - 3/31/00 1,750,656 F-01 WV 5/1/98-8/31/00 1,159,907 M-01 SC-Azeweb 9/1/98 - 1/31/01 170,000 LA-01 Goranboy Integrated 5/1/99 -11/30/00 2,396,512 First Round Subgrants Subtotal 13,814,782 H-06 ADRA 5/1/00-10/31/02 1,200,000 H-07 IMC 5/1/00 -10/31/02 1,097,911 H-08 IRC 5/1/00 -10/31/02 1,200,000 H-09 Pathfinder 5/1/00-10/31/02 599,745 H-10 ADRA-CDC Survey 10/25/00-12/31/01 149,615 H-10A ADRA 10/25/00-12/31/01 80,533 E-06 CHF 5/1/00-10/31/02 1,000,036 E-07 Save the Children 6/1/00-10/31/02 1,451,169 E-08 ACDI-VOCA 6/15/00-6/14/02 1,200,000 E-09 ADRA 11/1/00-10/31/02 882,005 CD-01 Save the Children 5/1/00-10/31/02 1,409,278 CD-02 World Vision 7/3/00-6/30/02 899,715 CD-03 IRC 8/1/00-10/31/02 1,100,000 SII-01 CHF 01/15/01-10/31/02 4,999,793 UN Funds -80,533 ICD-01 Save the Children 07/01/01-10/31/03 2,200,000 Food Program World Vision 06/01/02-01/31/03 75,000 ICD-02 IRC 01/01/01-10/31/03 2,799,984 2nd Round Subgrants Sub-total 22,264,251 Program Total 36,079,033 6.2 PROGRAM EFFECTIVENESS Mercy Corps has mounted a coherent and comprehensive range of programmatic interventions highly relevant to USAID’s Strategy under S.O. 3.1 and the Intermediate Results that support it. FINAL REPORT 36 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The AHAP umbrella has met or exceeded the mandate set forth by USAID in the original Cooperative Agreement. 6.2.1 Achievement of USAID Objectives Every grant-funded project, regardless of its sectoral focus, has as its center an organizational element that seeks to build sustainable capacity in target communities. Though the approach to community organization varies slightly from grantee to grantee, the common denominator is some form of community committee that identifies needs, manages activities designed to meet them, and is trained to repeat the process for other needs. This approach responds to IR 3.1.1.2-Communities Organized to Address Self-Defined Needs. Likewise, it is one of the key ways of achieving IR 3.1.1-Vulnerable Communities Better Able to Meet Their Own Needs. The substantive content of grantee activities, mostly health and economic development, including microfinance programs, as well as school and road construction, responds to achieving IR 3.1.1.1- Increased Access to Economic Opportunities and Support Services and IR 3.1.1.3 Communities Have Access to Better Quality Services. Under Mercy Corps, AHAP has evolved effectively from direct service to transitional programming. The AHAP grantee portfolio includes a mix of prominent and respected U.S. PVOs with acknowledged expertise in humanitarian and transitional programming. This broad base of providers was a conscious aim of USAID’s design. The mix and the high level of coordination among partners are two of the most valuable aspects of the umbrella. It is safe to say that the quality of programmatic interventions under AHAP is as relevant to USAID’s strategy and to the needs of target populations as the state of the development art permits. Most AHAP activities are located in the “IDP belt” and include IDPs as participants. In most instances there is some IDP membership on community committees. The AHAP umbrella has implemented an integrated approach to programming, bringing the full panoply of services in economic opportunity, health and community development to bear. This is reflected in grants to SCF and IRC for integrated activities in Central and Southern Regions, respectively, and in its encouragement of programming in clusters of communities. Mercy Corps claims to be dedicated to programmatic excellence, to learning, and to advancing the quality of development practice. Demonstrably, this is the case in AHAP, where these institutional values have dictated its approach. It is fair to say that in its execution, the AHAP umbrella under Mercy Corps has given rise to very strong programmatic interventions and to innovation that has broken new ground in Azerbaijan and allowed adaptations of proven methodologies to the country’s unique context. In some cases this commitment to quality has meant the Mercy Corps program has exceeded the expectations of USAID to the benefit of the program and its target populations. Whether the attendant costs of this approach are necessary in future S.O. 3.1 programming must be weighed by USAID. However, in light of USAID’s view that S.O. 3.1 programming is the “backbone” of its assistance in Azerbaijan, the quality that derives from the umbrella approach is a good investment. FINAL REPORT 37 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 6.2.2 Mobilizing Activities The AHAP umbrella has been extremely effective in mobilizing appropriate programming quickly. AHAP I RFAs were issued in March 1998, one month after Mercy Corps was awarded the agreement to manage the program. The second round of RFAs was issued for AHAP II in a similarly timely fashion. Proposers are given approximately six weeks from RFA issuance to submission deadline. The RFAs themselves are clear and complete, providing specific evaluation criteria, eligibility requirements and background information, including an appropriate sector strategy prepared by Mercy Corps. There are four such strategies: Community Development, Economic Opportunity, Health, and Social Investment Fund. 6.2.3 Selection Grant selection is conducted in-house by Mercy Corps. Proposals are reviewed and scored by a panel against the evaluation criteria explicitly set forth in the RFA. USAID is not involved in the initial review but participates in a review of short-listed proposals that “validates” the short-list. Following validation, the panel makes its final recommendations, which are referred to USAID for approval. Unsuccessful bidders received written comments on their proposals and may have face￾to-face debriefings if they wish. The review panels usually comprise Mercy Corps/Azerbaijan staff and one or more Mercy Corps headquarters staff and always the Mercy Corps/Azerbaijan Chief Financial Officer. The AHAP Chief of Party does not participate on the review panel so that he can more objectively manage the short-listing and validation process. The fact that Mercy Corps is so central to decision-making raises questions of conflict-of-interest, particularly in light of the trust issues discussed below. Participation of Mercy Corps headquarters staff is particularly questionable. They are in effect making decisions about their competitors while, at their headquarters, being centrally involved in bid decisions and proposal preparation on projects in direct competition with the AHAP partner organizations elsewhere in the world, and even in Azerbaijan. There is no evidence that Mercy Corps has acted in bad faith, but the appearance of the process unnecessarily opens the organization—and USAID—to criticism. The Evaluation Team heard no real allegations of unfair selection but did by the nature of the evaluation, talk only to successful bidders. Many umbrellas, including PVO-NIS, the original Russian NGO umbrella managed in the early 90’s by World Learning and the Institute for Sustainable Communities’ current Replication of Lessons Learned project, also in Russia, use selection processes that minimize the role of the umbrella manager in decisionmaking. This approach creates both the appearance and reality of an entirely objective process. The usual way to do so is to employ paid outside experts to review and score proposals. FINAL REPORT 38 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 6.2.4 Monitoring Monitoring, conducted quarterly, occupies about 50 percent of staff time. It is a key element of Mercy Corps’ approach to managing the AHAP umbrella. It is inextricably combined with a technical assistance function, which justifies Mercy Corps’ management structure organized around experts in the three technical sectors to which most of AHAP’s programmatic activity is devoted. It should be noted that ongoing technical assistance to partners is not a part of the USAID mandate. While it is reflective of Mercy Corps’ commitment to excellence and was a basic part of its AHAP proposal to USAID, it has not always operated effectively in practice and is viewed by some as redundant in light of the experience and technical expertise of grantees. Mercy Corps provides each grantee a detailed Annual Monitoring Plan outlining proposed dates of monitoring visits and broad topics. (See Annex E for a sample Annual Monitoring Plan). Perhaps the most original approach that Mercy Corps has taken is that from the outset, it has monitored on a geographic basis rather than grantee by grantee. The monitoring inquiry involves all the providers in a given area and site visits are jointly with all concerned organizations. Providers in the area meet to determine where site visits will take place. This approach has caused providers in each geographic area to think of the development challenges as a whole, not on the basis of their individual programs. Throughout most of the program, Mercy Corps’ sectoral teams have conducted monitoring of their respective programs separately. Within the last year, Mercy Corps also has moved to a model whereby sectoral teams are integrated and conduct each visit jointly. This, then, is the full meaning of “integrated” monitoring as defined by Mercy Corps: integrated at the field level in that the spectrum of interventions is reviewed as a whole and integrated at the oversight level in that all technical activities are likewise viewed through a single lens. Of all of the factors that have encouraged grantees to work on a coordinated basis, this undoubtedly is the most significant. It is a significant innovation; so significant in fact that it should be considered a model by USAID for encouraging interagency collaboration. Program-by-program monitoring interviews themselves are relatively free-form conversations based initially on proposed workplans and timelines. Within approximately two weeks of the monitoring visits, each grantee is sent a letter outlining issues to be discussed. These issues are the points of departure for subsequent visits. There is a joint debriefing, or “wrap-up” in Mercy Corps parlance. In deference to concerns about the time consumed by monitoring, wrap-ups with individual organizations were scrapped a few months ago. However, there was demand by some staff, especially local staff, for individual sessions, as well, so individual sessions are now optional at the discretion of the grantee. It is not without drawbacks of course. The numbers of people involved in site visits can seem to recipients more like a royal procession than a monitoring event with perhaps 20 people and a cavalcade of vehicles descending on them at once. A number of grantees indicated that Mercy Corps monitoring is burdensome because of its frequency combined with the depth and duration of FINAL REPORT 39 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation visits that, accounting for required preparation and the visit itself, can consume the better part of two weeks. Monitoring was reduced to quarterly from every two months. On balance, the benefits far outweigh the drawbacks. Benefits include the intelligent and thoughtful assessment of AHAP interventions in their totality, increasing coordination and the opportunity provided for all staff levels of provider organizations to participate and to learn from the experience. While the process is excellent, even quarterly monitoring of such mature and experienced organizations seems excessive, especially considering the depth and intensity with which it is carried out. Semi-annually would appear to be adequate and a more judicious use of time and resources. Monitoring of completed projects has been left to individual implementers and there has been no formal mechanism for including this important data in forward planning. Moreover, such post￾project monitoring has been spotty and sometimes not carried out at all because projects have ended or other imperatives have intervened. This suggests that consistent post-project monitoring should be the job of S.O. 3.1 managers, whether USAID or an umbrella manager. Spot checks, designed to measure independent activity after the end of formal engagement with implementers and the continued cohesiveness of community groups, should be made at three, six and 12 month intervals. The data so derived should be an important part of determining the nature of future activities, their locations, the need for follow-up training and especially the composition of cluster activities. 6.2.5 Program Reporting In addition to the monitoring visits, each grantee provides reports semi-annually against a narrative format devised by Mercy Corps. The format is modeled on the report that Mercy Corps makes to USAID and includes sections on progress, problems, challenges and lessons learned. In addition, each provider reports against the USAID Performance Monitoring Plan (PMP) indicators relevant to its program. Mercy Corps combines these reports in its own semi-annual report to USAID. A typical Mercy Corps report runs upwards of 40 pages of narrative, a few photos, charts detailing PMP achievements and financial information. There is a summary section of four to five pages and separate narratives of similar length for each of the sectoral foci. All of this information is provided on a program-wide basis. In addition, a separate section includes half-page “success stories” that feature individual grantees. Overall, the reports give a good indication of the state of implementation of AHAP programs and future plans and are reflective of Mercy Corps’ management competence. They are clear, well￾written and present relevant data in an accessible fashion. Reports are not shared with grantees. 6.2.6 Quality of Data Because of the broad base of programming in AHAP, involving several sectors, Mercy Corps reports to USAID against 150 separate indicators. At present, this seems unavoidable, given the FINAL REPORT 40 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation nature of the current PMP, but any discussion of reporting must take into account the limitations of that document and the circumstances in which it was formulated. Most Intermediate Result (IR) level indicators are output type, which was appropriate to the constraints of Section 907 and to the overall transitional nature of the US foreign assistance strategy in Azerbaijan at the time the plan was designed. By definition, USAID was not in a position to mount a more far-reaching program or to devise more qualitative indicators. Within these parameters, data provided by AHAP implementers is in itself good. Its meaningfulness and value to forward decision-making, particularly to a development environment no longer constrained by Section 907, is open to question. Undoubtedly, as USAID defines its next strategy in a more hospitable environment, it will be in a position to establish more reflective of what it is hoping to achieve in Azerbaijan and, it is hoped, fewer of them. 6.2.7 Financial Reporting Grantees provide quarterly financial reports to Mercy Corps which trigger drawdowns of funding in much the same way that Mercy Corps itself reports to USAID for the same purpose. Funds are sent to grantee organizations according to their preference, sometimes to the home office and sometimes to the field office. Broad financial issues, particularly rates of expenditures, are included in monitoring discussions. Specific issues usually are handled between the Mercy Corps Chief Financial Officer and financial staff of respective grantees, in the field or with their headquarters as appropriate. At one point, Mercy Corps was using drawdowns as a management tool, tying them to performance. This practice was ended at the request of the USAID Coordinator and advances now are made more routinely as is appropriate in a grant relationship. 6.2.8 Compliance Compliance with USAID regulations and procedures and with the terms of contracts with Mercy Corps is the responsibility of the Chief Financial Officer. In general, this compliance oversight seems to be carried out well. A few grantee informants grumbled about specific instances where Mercy Corps had provided erroneous guidance about allowability of certain actions but these can be seen as isolated, and not unexpected, incidents. Large management issues, such as the suitability of grantee management and staffing plans as they relate both to programmatic and financial issues are handled at the proposal stage. On several occasions, grantees have been asked to strengthen the caliber of management staff or levels of effort as conditions of grant award. Mercy Corps requires copies of A-133 audits and reviews them, especially in instances where grantee operations in Azerbaijan have been singled out for comment. In these cases, grantees are required to address the issues in question. Broad compliance requirements are spelled out in Mercy Corps’ agreements with grantees. Specific compliance oversight is carried out in conversations between the Mercy Corps Chief Financial Officer and grantees in the context of questions about allowability as they arise. Until recently, Section 907 generated many questions about allowability of both activities and FINAL REPORT 41 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation expenditures. Other frequent issues in the AHAP program are procurement, especially of construction materials, source and origin requirements given Azerbaijan’s proximity to Iran, allowable travel, and consultants. Allowability and handling of in-kind matching contributions also generate questions from grantees and rulings by Mercy Corps. Consultant allowability and travel seem to be two compliance areas that have generated confusion, confirmed by Mercy Corps itself and by grantees. Generally speaking, in both Cooperative Agreements and grants, USAID regulations allow these without prior authorization when they are within budgeted amounts and their purposes are substantiated in narratives. Approvals for consultants and travel have been contentious. Grantees of course hold Mercy Corps responsible, while Mercy Corps points to USAID expectations. Since Mercy Corps has overall fiduciary responsibility for grants to partners, its caution is understandable, but the nature of the umbrella means that two layers of approval are required. The resulting delays, questions and discussion do not seem in the best interest of USAID, Mercy Corps and, most importantly, efficiency of program operation. While the degree of necessary budget substantiation is subject to interpretation from USAID Mission to USAID Mission and individual by individual, and often is, efficiency of operation would dictate that the most generous possible interpretation is the most desirable. Of course, source and origin and other requirements must be strictly adhered to. Overall, USAID did not indicate concerns about Mercy Corps’ ability to secure satisfactory compliance and there was no evidence that major problems have arisen in this area. The fact that some USAID functions, particularly contracting and financial, are located in Tbilisi leads occasionally to conflicting messages, but this did not seem to be a serious problem and is one that probably is encountered at all levels of managing the Azerbaijan USAID office, as it is in any situation where complex management arrangements pertain. 6.2.9 Use of Data for Decision-making Two striking impressions emerge from the team’s review of how decisions about program direction are made in AHAP. The first is that Mercy Corps has wide latitude in determining the particulars of how USAID’s S.O. 3.1 Strategy is implemented. After initial involvement in design and project launch, there has been little programmatic tinkering by USAID beyond approvals of RFAs and grant awards. This is understandable in light of the small staff in the Azerbaijan office and the multiplicity of demands placed on it. The second impression is that despite the copious collection of information and assiduous reporting, the evolving program direction emerges more organically from Mercy Corps management and experience than from a systematized effort to analyze and interpret data. This arrangement raises the concern that, while Mercy Corps’ institutional memory is invaluable and likely for all kinds of systemic reasons to be more stable than USAID’s, a few staff changes can damage it considerably. In some cases where positions in the Baku office were vacant for long periods, this has occurred. FINAL REPORT 42 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation This is not to say that there is anything wrong with program direction. It already has been established that within constraints of funding, the Azerbaijan context and Section 907, Mercy Corps has mounted an excellent program. Nonetheless, USAID’s relatively small role seems a missed opportunity, both from the standpoint of fine-tuning its work in S.O. 3.1 and lessons that would inform its larger assistance program in Azerbaijan. USAID and umbrella manager review of performance data, more formally and regularly, perhaps in consultation with partners, as a prelude to decision-making would be more desirable. Mercy Corps and USAID have used a detailed set of Communication and Management Protocols to guide their interactions throughout AHAP implementation. Such protocols should be revised to include reviews and similar mechanisms that would enhance USAID’s involvement in forward program planning. This will be especially relevant in the future when new and more qualitative performance indicators have been defined. 6.2.10 Coordination The most striking and valuable aspect of the AHAP umbrella—and of Mercy Corps’ management of it—is the collaboration that exists among partner organizations. Mercy Corps has created an atmosphere that, in innumerable ways, is hospitable to collaboration, the most notable of which is the integrated approach to monitoring. Collaboration to the degree it was observed by the Evaluation Team rarely is found and greatly increases the impact and effectiveness of the AHAP partnership. Examples of collaboration are numerous. There is frequent referral among partner organizations when one organization has an expertise that another does not. The Social Investment Initiative (SII), which supports larger Babi Hospital, in Fizuli District, is the nonpareil example of community self-help and AHAP partnering in USAID’s health portfolio under S.O. 3. From a number of perspectives, it’s one of those projects when everything went “right.” Two surgeon brothers who had worked in an area army hospital during the war with Armenia returned again to practice to there in 1996 after the Armenians retreated from the district. The community of Babi made an old store available which the doctors began to convert into private practice medical facility using their own resources but it quickly proved too small and lacked water and appropriate sanitation facilities. With help from IMC, a six member Community Health Management Committee (CHMC) was formed. With a little financial support from IMC, a new hospital was begun by the community with its own funds and labor but resources soon ran out. CHF awarded the community a $20,000 CEEOP grant, matched by the community with $9,800 in cash and $3,000 in kind and an IMC contribution of $3,000. The completed hospital, with potable water, sterilization equipment, medical waste disposal facility, an operating theater, recovery room, wards and an ambulatory primary care clinic received more than $100,000 worth of medicines, supplies and equipment from IMC and the State Department, and IMC has provided training to the medical staff. It is the only MOH licensed private facility of its kind outside of Baku. The facility is owned by the community and leased to the Babi doctors’ association. Lease payments take two forms: cash payments, used by the community as contributions to the facility’s maintenance fund, and continuing free care for the community’s population of approximately 1,100. It serves not only the Babi population, but on a fee basis, more than 10,000 from the surrounding area. FINAL REPORT 43 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation infrastructural investments in communities that have demonstrated the ability to carry them out, designed to reinforce community capacity in self help, also is a natural incentive for individual grantees to work with CHF, the SII manager, in order to scale up their projects. In the Central Region, CHR, IRC, SCF and ACDI/VOCA publish Taraggi, a joint Azeri language development newsletter. It displays all their logos and the evaluation team saw it posted or otherwise displayed in a number of communities. Its ready and wide availability attests both to its value and the diligence of the partners in distributing it. The most important aspect from the point of view of the umbrella is that this was a spontaneous initiative of the cooperating partners, not one imposed by the umbrella, suggesting that Mercy Corps has successfully created an collaborative environment that goes beyond its own initiatives. The evaluation team attended the National Community Development Conference held in Baku for 11-13 October. Some 300 AHAP partners and community representatives spent two days discussing aspects of community development. Subsequently, as the team traveled around the country, community member after community member mentioned the conference and its value to them. The opportunity for community people to reinforce, through discussions with others in a similar situation, the lessons they have learned through AHAP participation at home and to build networks and make connections is incalculable. The Community Development Conference was the third such conference sponsored by Mercy Corps. A conference devoted to health was convened in 1999 and one devoted to microfinance in 2001. In addition, AHAP encourages smaller, less elaborate regional meetings on various topics related to the program as well as “cross visits” where staff and participants from one project visit another project to discuss lessons learned. One recent example was an economic opportunity cross visit in Naxcivan, Barda, Imishli and Ganga involving SCF, ADRA, CHF, and IRC. FINCA, which is not a part of the AHAP partnership, also participated in the visit, indicating the value of these events to participants. The visits include site visits to projects as well as discussions and synthesis sessions. In addition, Mercy Corps holds regular sectoral meetings at which program staff of various partners meet to discuss common problems, shared issues and current trends. For much of his tenure, the Chief Financial Officer has chaired meetings of financial personnel of grantee organizations. Mercy Corps also publishes a national level newsletter, the AHAP Bulletin, which describes general program accomplishments, interviews with staff, focus features on individual programs and lessons learned. Mercy Corps also issues a monthly newsletter devoted to economic opportunity issues. Capacity building of local staff is an important side benefit of partner collaboration in AHAP. The many vehicles that encourage coordination provide opportunities for local staff to interact, discuss problems and issues and learn about new and effective approaches. Building local staff capacity always is valuable but often, it is done in conjunction with local NGOs to carry on the work of external organizations. In Azerbaijan, where the enabling environment for local NGOs has been hostile, building capacities of local individuals is doubly valuable. FINAL REPORT 44 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 6.2.11 Trust Perhaps the most nettlesome issue for AHAP and the area in which Mercy Corps has been least successful is creation of trust between grantees and itself. Mistrust is to some degree inherent in a system where an umbrella is managed by a peer organization, particularly in the regrettably competitive environment of U.S. PVOs. However, there is little evidence that Mercy Corps has found ways to deal proactively or sensitively with the issue. At the root of the matter is the fact is that by virtue of its selection responsibilities, Mercy Corps has access to a variety of proprietary information about grantees who are its competitors. This encompasses cost structures, including NICRAs, management plans and budgeting practices as well as implementation methodologies. While organizations undoubtedly exaggerate the unique nature of their programmatic approaches, the fact remains that Mercy Corps has a valuable store of information about its competitors as well as how, and how well, they write proposals. Some grantees believe that Mercy Corps could—or does—use this information to prepare its own proposals. However unfounded these suspicions, the perception was aggravated by Mercy Corps’ decision to compete successfully against its own grantees in Azerbaijan for AID/Washington￾funded programs in Child Survival and Business Development. While USAID/Azerbaijan urged against their approval in Washington, the contracts were awarded. In at least once instance Mercy Corps was said to have attempted to force a grantee to divulge information about the programmatic approaches it was using in its work under AHAP. While this was likely in the interest of improving overall performance of AHAP grantees and of maximizing impact, it seems heavy-handed, especially in circumstances already characterized by mistrust. One very useful firewall would be to remove itself entirely from grant award decision-making, using outside experts instead. Erecting a barrier between proprietary information and the Mercy Corps home office marketing functions would send an important message about transparency to AHAP partners. Because of the atmosphere, some grantees believe that Mercy Corps tries to take credit for their achievements and cite as an example the fact that semi-annual reports to USAID are not shared with them. To be sure, the reports, except for success stories, are written generically and do not credit specific grantees, nor at this level USAID is likely to be interested in individual providers. In actuality, sharing the reports more widely would not likely build trust, since Mercy Corps has a responsibility to be candid in its assessment of partner performance, and no partner organization would, on balance, welcome the widespread availability of such information. Three structural options for addressing the trust issue suggest themselves: abandoning the umbrella concept in favor of direct administration by USAID, confining eligibility for umbrella manager to for-profit institutions not in direct competition with PVO grantees, and living with the conflicts inherent in the present arrangement. A fourth, and most desirable, would be for the umbrella manager to exercise greater leadership in clarifying its efforts to prevent misuse of information. USAID also could take stronger steps to FINAL REPORT 45 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation discourage the appearance of conflict to the degree it legally can do so. One such measure would be including non-compete language in future RFAs. 6.2.12 Grantee Attitudes Grantee attitudes toward the umbrella were generally positive. All partners were very complimentary about the impact that coordination it has fostered on their own organizations and the quality of their work under S.O. 3.1. Recurrent negative comments included the conflict-of￾interest issue and time consumed by monitoring visits, though not the value of monitoring itself. A number of grantees questioned the value of the technical assistance function in the Mercy Corps design. At least one grantee admitted that it was easier to express negative views to Mercy Corps than to USAID directly and that Mercy Corps probably is in a better position to be responsive than USAID would be. All seemed comfortable with USAID’s accessibility as a “court-of-last-resort” for difficult issues. The evaluation team found the few recurrent criticisms valid; they are addressed as recommendations. 6.3 PROGRAM COST Mercy Corps has managed the umbrella very cost-effectively. Of a total six-year budget of $45 million it is anticipated that more than $37 million will have been awarded in grants, making the annual management cost approximately $1.3 million. Mercy Corps has a low indirect cost rate of 17 percent and is very frugal in its home office management charges. In addition, its salary structure and especially its expatriate allowance structures are significantly lower than USAID’s. Unquestionably, USAID has received good value from Mercy Corps’ management of the AHAP umbrella and several of the recommendations of the evaluation team will point to further modest savings in the current umbrella management costs. 6.3.1 Staffing Mercy Corps maintains an office on Magomayev Street in Baku with a staff of some 30 persons. Six are expatriates and include the Chief of Party, a Chief Financial Officer and a Program Director and three expatriate sector specialists in Health, Economic Opportunity and Community Development. Local staff report to these expatriate managers. (See Annex E for the Mercy Corps Organization Chart). The Chief of Party is responsible for overall program management, liaison to USAID, and contacts with other organizations. Responsibility for managing the AHAP program rests with the Program Director, who oversees the three expatriate technical specialists. In USAID’s original conception, the justification for sectoral expertise was to ensure technical competence in the grant award process. Mercy Corps has expanded this role to embrace an overt emphasis on technical assistance to partner organizations. In practice, the experts function as a team that manages monitoring, reviews reports and proposals and spearheads the complex set of activities that have yielded the high level collaboration among AHAP partners. FINAL REPORT 46 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation It is not clear that their role as experts per se, as opposed to their role in facilitating dialogue and highlighting problems and solutions has been a key factor in the success of the AHAP partnership. Most of the partner organizations have strong field-based expertise of their own, undergirded by dedicated technical components in their headquarters offices. USAID has excellent expertise of its own in Azerbaijan and Tbilisi as well as in Washington. Recruitment for Azerbaijan-based positions is not easy and there have been long periods when one or the other technical position was vacant. This has given rise to complaints by partners that they have to “re-educate” Mercy Corps experts when they finally are hired to fill vacancies and that the technical assistance layer is redundant. At present, the longest-serving technical expert has been in place for slightly more than a year. While there is value in the technical assistance role, the Mercy Corps role for providing it seems flawed. It is likely the greatest benefits of expertise have arisen not from the in-house experts, but from coordination among partners. Taking into account all factors and considering the current level of maturity of the AHAP partners and the fact that three expatriates significantly raise the umbrella’s cost, these criticisms seem justified. By now it should be possible for local personnel to head the umbrella’s sectoral units. Expertise for proposal selection and for focused expertise that may from time to time be required can be secured more economically through short-term consultancies. The umbrella manager then can focus its energies where they have proved to be most useful: fostering collaboration and information exchange and its integrated monitoring that serves to surface key areas where remedial attention is indicated. The umbrella should have the financial resources and authority to deploy outside consultants quickly and easily and should serve as a sort of clearinghouse for the purpose. 6.3.2 Cost Implication of Umbrella vs. Direct Management Almost certainly USAID’s costs would not be lower in managing the umbrella directly. In addition, there are internal issues of operations versus program budgets. USAID would require a staff of at least five to manage the current portfolio. It currently has two on board, the Humanitarian Response Program Specialist and a Program Development Specialist. Additional personnel would include a second Program Development Specialist-level position, TCN or local hire, and an additional staff person for contracts to complement the work of the existing Senior Acquisition Specialist. Additional administrative support would be required, as well. Conservatively, the cost of these additional positions would be $500,000, assuming they were available, on top of the cost of the current S.O. 3.1 staff complement. There also would be the added costs of coordination activities such as conferences, technical assistance and administrative intangibles like transportation, financial management, general administrative support and office space. One way to reduce the requirement of additional staff would be to reduce the number of providers. This could be done by letting one or two contracts for integrated programming in several regions of the country or for sectorally-focused activities. However, one of the undoubted values of AHAP and one which was a deliberate part of USAID’s design, is the synergy provide by a range FINAL REPORT 47 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation of highly-experienced players. Maintaining this broad-based portfolio in a direct management model would be difficult with a staff of five and impossible with a smaller staff. Direct management also would sacrifice the intensive oversight provided by the umbrella and the ongoing emphasis on coordination and collaboration, even with an in-house staff of five. The choice for USAID will depend to a large degree of internal priorities. Where do the activities now carried out under S.O. 3.1 fit in the larger scheme of USAID objectives? Do they support other, more important objectives or are they overriding? If the current S.O. 3.1 objectives— especially community development—are not central features of the USAID strategy, the intensive level of oversight provided by the umbrella may be redundant. What are the feasibility and desirability of adding internal staff? Since additional staff financed by the operations budget could be deployed in a variety of areas, are they best used on activities now carried out under S.O. 3.1? 6.3.3 Value-added Mercy Corps cites five “value-adds” provided through the AHAP umbrella. They are: • Coordination; • Information dissemination, information-sharing and learning; • Technical assistance expertise; • Program oversight; • Government relations. It might add a sixth, which has been to serve as the major engine for strategic planning and program evolution for S.O. 3.1. The AHAP umbrella under the stewardship of Mercy Corps has demonstrated that it does indeed provide these attributes to the program. A similar program managed internally by USAID would face two key obstacles in providing the same level of value￾added. First, Mercy Corps has a staff that is five or six times bigger than any conceivable USAID staff configuration and so can oversee the AHAP portfolio much more intensively. Second, as an official arm of the US government, USAID has less flexibility than a Cooperative Agreement holder like Mercy Corps and would be limited in its ability to respond as quickly and comprehensively as the umbrella. To guide USAID thinking in deciding between and umbrella mechanism or direct management, a brief discussion of the value-added of the umbrella approach is useful: COORDINATION It already has been established that AHAP has achieved enviable results in coordinating the activities of partner organizations, targeting them geographically and programmatically and in using RFAs, monitoring and information-sharing tools to move well beyond direct humanitarian assistance. Coordination has been a powerful factor in enabling a higher level of impact than would otherwise have been possible. It is doubtful that an S.O. 3.1 project managed internally would be able to encourage coordination as consistently as the umbrella manager has done. This FINAL REPORT 48 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation handicap would be mitigated to some degree in the future by the fact that the ethic of collaboration already has been established and would be unlikely to wither entirely. INFORMATION DISSEMINATION, INFORMATION-SHARING AND LEARNING The mechanisms that Mercy Corps has used for information sharing—sectoral meetings, newsletters, integrated monitoring, seminars, and conferences—have been the major tools for coordination and also have enabled partners to learn valuable methodologies and approaches from each other. One excellent example is the cross-fertilization that has occurred with respect to cost recovery in health programs. Each of these information mechanisms requires planning and logistics on a scale it would be difficult for USAID to provide. Major undertakings, like the recent National Community Development Conference no doubt would require contracting an event planner to carry out. However, it is possible that World Learning’s START participant training program could pick up some of these responsibilities. TECHNICAL ASSISTANCE EXPERTISE The umbrella unquestionably has created a program dynamic whose quality and impact are greater than the sum of its parts. Credit for this goes equally to USAID, whose initial design anticipated this result, to the partners and to Mercy Corps. It is not clear that Mercy Corps’ design, which emphasizes on-staff technical expertise, has contributed significantly to program quality. It is just as likely that the active collaboration within the program and the sharing of information have been the important factors, coupled with the experience and expertise of partner organizations. In the arena of technical assistance, USAID could probably achieve the same results as Mercy Corps has done through the judicious use of its own in-house expertise and the availability of consulting funds in grantee budgets and through Mission-financed consultancies where necessary. PROGRAM OVERSIGHT USAID informants themselves have indicated that, owing to inherent staffing limitations, the intense level of oversight and monitoring that has characterized the Mercy Corps umbrella would not be possible in a directly-managed program. The question, then, is whether the level of oversight afforded by the umbrella is necessary to program impact, at least as future implementation and transition of humanitarian assistance programs go forward. In any case, diminished intensity of oversight would be compensated for to some extent by USAID’s more intimate involvement with the program and with being integrated more fully into the daily operations of USAID. A side benefit should be that USAID would have more ready access to lessons learned and to trends and developments in the field from which, under the umbrella arrangement, it perforce is more distant. GOVERNMENT RELATIONS Mercy Corps has played a valuable role in relations to the GOAJ, particularly during the period when Section 907 was in place. It has been similarly valuable to the partnership, both on substantive programmatic issues and on bureaucratic matters like taxation. It can continue to be useful in this respect, perhaps to even greater effect now that direct assistance to the government is FINAL REPORT 49 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation possible. It can interact with the GOAJ at all levels, bringing with it the implied weight of all the AHAP partners and has the ability to serve USAID’s interests by serving as an intermediary on matters of mutual concern where an official US entity would have to exercise more caution. On balance, this aspect of the umbrella mechanism will continue to be an important resource to USAID in managing S.O. 3.1. STRATEGIC PLANNING While AHAP has reflected a conception set forth by USAID in its RFAs, subsequent innovations and evolutions, including the cluster approach, SII and integrated projects have emerged from Mercy Corps’ experience in the field. Regardless of whether USAID opts for an umbrella or direct management, the Evaluation Team recommends closer involvement by USAID in fine-tuning program implementation. This will be stronger coordination of its complete Azerbaijan assistance program and better access to the lessons derived from implementation of S.O. 3.1. With direct management, USAID would lose the benefit of the undoubted wisdom of senior Mercy Corps staff, but on balance there appears to be an even tradeoff. In either case, USAID should play a larger role in future strategic planning, as it has stated that it will to do. 6.4 SUMMARY Ultimately, the overriding advantage that USAID derives from the umbrella and the one it is least likely to replicate in direct management is the wide mix of providers, which brings an important dynamism to the S.O. 3.1 program. Another advantage is the intensive management it has afforded. More intensive management has in turn yielded excellent coordination, and its by￾product, strong local staff capacity-building. Finally, it is no reflection on USAID that it has less flexibility to respond quickly and creatively to a changing programmatic context than an organization like Mercy Corps that operates outside the USAID system. Together, these advantages have enabled Mercy Corps to create a program of uncommonly high quality. It has done so at a cost roughly equal to the cost of internal management. To replicate it internally would require, in addition to added staff, issuance and award of at least six separate and different RFAs, perhaps more, and place a heavy management burden on the USAID office overall. 6.5 KEY FINDINGS • The wide mix of implementers and the high level of coordination among partners are two of the most valuable aspects of the umbrella; • Mercy Corps has mounted a coherent and comprehensive range of programmatic interventions highly relevant to USAID’s Strategy under S.O. 3.1 and the Intermediate Results that support it; • The AHAP umbrella has been extremely effective in mobilizing appropriate programming quickly; FINAL REPORT 50 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation • AHAP technical assistance has not always been effective in practice and is viewed by some as redundant in light of the experience and technical expertise of grantees, nor is it a part of the original USAID mandate; • Perhaps the most original approach that Mercy Corps has taken is that from the outset, it has monitored on a geographic basis rather than grantee by grantee; • While the monitoring process is excellent, even quarterly monitoring of such mature and experienced organizations seems excessive; • The quality and reliability of monitoring data is good; its meaningfulness and value to forward decision-making are open to question; • Monitoring of completed projects has been left to individual implementers and there has been no formal mechanism for including this important data in forward planning. • In general, the compliance oversight seems to be carried out well by Mercy Corps; • Mercy Corps has had wide latitude in determining the particulars of how the USAID’s S.O. 3.1 Strategy is implemented; • Evolving program direction emerges more organically from Mercy Corps management and experience than from a systematized effort to analyze and interpret data; • USAID’s small role in designing evolving program strategy seems a missed opportunity, both from the standpoint of fine-tuning its work in S.O. 3.1 and lessons that would inform its larger assistance program in Azerbaijan; • Perhaps the area in which Mercy Corps has been least successful is creation of trust between grantees and itself; • The fact that Mercy Corps is so central to grant award decision-making raises some serious questions of conflict-of-interest; • Mercy Corps has managed the umbrella very cost-effectively and USAID unquestionably has received good value; • Almost certainly USAID’s costs would not be lower in managing the umbrella directly and there are internal financial resource issues of operations versus program budgets; • The wide mix or providers is the aspect of the umbrella that USAID is least likely to be able replicate in direct management. FINAL REPORT 51 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 6.6 RECOMMENDATIONS • All things being equal, a continued umbrella is preferable to direct management by USAID. The major reasons are the more intensive oversight and monitoring that an umbrella can provide and the richer mix of implementing organizations it affords. USAID’s ultimate decision will of course depend on considerations about its overall assistance program in Azerbaijan, available resources and internal management capacity. • Should USAID opt to continue the umbrella mechanism, it should institute a system of formal review of program progress with the umbrella manager. The review would address issues such as achievements to date, possible refinements, lessons learned and future planning. The review should be conducted semi-annually, using the umbrella manager’s report as its basis. • In a future umbrella, staffing should be reduced by approximately 10 persons. The umbrella itself should have no more than three expatriates (a Chief of Party, a Program Director and a Chief Financial Officer). De-emphasis on internal technical assistance and reduced monitoring frequency would enable these reductions. • In keeping with these reductions in staff, the umbrella manager should focus its energies where they have proved to be most useful: fostering collaboration and information exchange and its integrated monitoring that serves to surface key areas where remedial attention is indicated. • To fill the gap occasioned by a reduction in full-time internal technical assistance capacity, the umbrella manager should budget funds for a clearinghouse for short-term technical assistance, as required by grantees, and as a part of the clearinghouse should maintain a database of consultants with appropriate expertise should grantees require assistance in locating the required skills. Similarly, grantees should be encouraged to budget for consulting in their respective budgets. • To complement the revised technical assistance model, USAID should make every effort to make use of consultants as quick and easy as possible. • Any umbrella manager should continue to employ the highly successful integrated monitoring system developed by Mercy Corps. Monitoring frequency should, however, be decreased to semi-annual. Mercy Corps’ monitoring has been a very effective tool in identifying programmatic weaknesses and needs for outside expertise; the umbrella manager should continue to make this aspect of monitoring a priority and should draw on the technical assistance clearinghouse to meet needs it identifies. • Consistent post-project monitoring should be the job of S.O. 3.1 managers in the form of follow-up spot checks on the sustainability of project interventions. This information would inform the nature of future activities, their locations, the need for follow-up training and especially the composition of cluster activities. FINAL REPORT 52 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation • Both USAID and its S.O. 3.1 programs will benefit from development of a new USAID strategy that reflects the current development environment in Azerbaijan. The new PMP should focus on qualitative information relevant to the fundamental objectives the S.O. 3.1 program is aimed at achieving and should have many fewer indicators than the 150 output￾related indicators in the current plan. • On balance, a PVO umbrella manager is the logical manager for a future umbrella, both for reasons of cost and of culture. The shared cultures of the PVO Mercy Corps and its PVO grantees have yielded positive results that a for-profit umbrella manager would be unlikely to duplicate easily. • USAID should take whatever legal measures are available to it to discourage the umbrella manager from undertaking implementation activities in Azerbaijan while it holds the umbrella agreement. This would include non-compete language in any future RFAs for umbrella management. • The umbrella manager should spell out the means by which it would minimize both the appearance and reality of conflict-of-interest arising from the knowledge of proprietary information at its disposal arising from its management of grants to other US PVOs in the umbrella partnership. • The umbrella manager should remove itself entirely from grant award selection and use disinterested experts for review of proposals and recommending to USAID. This does not mean that it would not participate in the review process in an advisory capacity that took advantage of its knowledge of the program and its requirements and of strategic planning objectives for the program. USAID should, of course, retain final approval authority for grants. FINAL REPORT 53 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation SECTION 7 FINDINGS AND RECOMMENDATIONS The evaluation statement of work provides the following: “The purpose of this evaluation is to review S.O. 3.1, its associated activities, and the performance of AHAP as a management mechanism. The findings and recommendations will inform Mission management and the implementing Partners how effective the current program has been in meeting the strategic objective and where and how possible adjustments could be made to improve program implementation and impact.” Over the past two years, the Mercy Corps umbrella managed a program of, on the average, about 14 subgrants to eight Partner INGOs. These subgrants were for programs in the four USAID conflict-affected areas of Central and South IDP Belt, the Urban area near Baku, and Naxcivan. . 7.1 EVALUATION QUESTIONS The evaluation statement of work posed a series of evaluation questions associated with a review of S.O. 3.1, Reduced Human Suffering in Conflict-Affected Areas, to be addressed by the Evaluation Team. Key findings and recommendations in response to these evaluation questions are presented below. 7.1.1 Critical Humanitarian Needs Partner subgrantee personnel in the field have indicated that, to their knowledge, the critical humanitarian needs of emergency food and shelter are generally available to all IDPs. There are, however, pockets of greater need, e.g. in Naxcivan and in many communities with deteriorating infrastructure and a lack of economic opportunities. In a few areas, available food may be of poor nutritional quality. While medical services, at some level, are available, the quality of and access to health care and pharmaceuticals vary widely. Housing conditions range from unheated railroad boxcars to newly constructed individual family houses with electricity and drinking water. In the conflict-affected areas visited by the Evaluation Team, local non-IDP residents as well as the IDPs in these communities have need for medical services, electricity, safe drinking water, and improved roads. According to community and IDP leaders that were interviewed, in addition to health and infrastructure needs, school buildings and community centers were high priorities. These were closely followed by agricultural economic opportunity needs (irrigation, salt water drainage, food processing), youth programs (sports fields and gymnasiums), and ceremonial sites for weddings and funerals. FINAL REPORT 54 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 7.1.2 Transfer of AHAP Economic Opportunity Projects to the Mission S.O. 1.3, Accelerated Growth of Small and Medium-Sized Enterprises in Targeted Areas. The Evaluation Team visited several economic opportunity programs that were implemented under S.O. 3.1 in the "IDP belt" of Central and South Azerbaijan. The development of these projects, primarily for irrigation water and drainage, relied heavily on community mobilization efforts to organize community members who then focused on community needs and priorities. In most of the sites visited, the communities developed health clinic and/or school projects prior to undertaking economic opportunity projects. This "integrated" approach would seem to indicate that separation of community development and economic opportunity from humanitarian oriented health, education, youth, and infrastructure (water, electricity, roads) projects may be counter to the expressed community priorities. Economic opportunity projects were conducted under SII supported community groups that designed and implemented projects of community-wide interest and impact. These projects were in irrigation and agricultural product processing. Removing these kinds of economic opportunity projects would weaken the community mobilization and community development efforts that have been shown to have an enhanced effect through addressing economic opportunity concerns. The microfinance projects of Save the Children (SC) and, in Naxcivan, ADRA on the other hand, have not demonstrated that they contribute significant synergism in support of either community development or health programming. Such projects and grants could be moved to direct USAID management under S.O. 1.3, Economic Development. In making that move, additional regional responsibility will result for USAID personnel who manage S.O. 1.3 since these microfinance projects are currently in Naxcivan and the Central Region of the IDP Belt. Loan projects conducted by SC in the Central region of the IDP belt and by ADRA in Naxcivan appeared to have supported borrower enterprises and loan objectives with relatively few failures (an estimated failure rate of 10% by the Evaluation Team). Repayment rates were at approximately 98%. The conclusion from Evaluation Team visits to these Partner selected sites is that the loan projects were working as planned. Three focus groups, composed of persons with no or very little contact with AHAP programming, were conducted. Two were in the Central region and one was in Naxcivan. Individuals in these groups, in addition to identifying community needs, stressed the lack of economic activity in their areas. The communities visited in the urban area near Baku were noticeably in need of additional economic activity. The obvious conclusion from these focus group sessions and by observation in the urban target area is that economic opportunity projects are widely needed. 7.1.3 Program Effectiveness The Evaluation Team collected information regarding the implementation and effectiveness of subgrantee activities under the three major program areas (community development, health, and economic opportunity) with respect to achieving USAID objectives and intended results. FINAL REPORT 55 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The Evaluation Team observed many project sites where activities in all three major program areas have been successfully attained to the planned objectives of those communities. Accordingly, to the extent that USAID is aligned with community priorities, USAID objectives and intended results have been achieved in these cases. All three program areas, in the sites that demonstrated an integrated and sub-grantee coordinated approach, were viewed as being necessary to addressing the S.O. objectives but in some cases were not yet sufficient in their duration or scope. Where Partner integration was occurring, all three areas were mutually effective and dependent upon each other to achieve the level of effectiveness noted. These successful endeavors also resulted in community organizations that stated that they were sufficiently confident in their own abilities to undertake new projects with and, if necessary, without donor support. The Evaluation Team noted a major constraint to achievement of community goals. The additional resources available through the USAID subgrantees had a high impact on community growth, satisfaction and realization of subsequent projects. Inadequately supported efforts failed to achieve their maximum potential. The one case where the USAID financial support was in the range of $10,000 and not sufficient to the several projects that were undertaken, nonetheless, was assistance that was greatly appreciated by the community. They were prepared to move onto other projects despite the fact that the school roof still wasn’t replaced and leaked. The constraints varied from project site to project site. In one case, support/approval of a community project was denied by the Executive Com, the regional implementation agency of the national government, and required appeal to higher governmental levels. In all other cases support including a mix of encouragement, labor, technical assistance, materials, and direct financial contribution was provided by Municipal and/or Ex Com officials. A few mistakes were made in terms of underestimating demand for processed agricultural projects and steps were being planned to increase production capacity. The Evaluation Team sees an opportunity for USAID to further engage government officials at all levels. 7.1.4 Subgrantee Objectives and Meeting Needs of Targeted Beneficiaries In the cases observed by the Evaluation Team, objectives were largely well defined for each subgrantee. According to beneficiaries (where community groups defined the needs and priorities), the sub-activities were more than meeting the expressed needs and expectations. At every community meeting, the beneficiaries expressed their gratitude for the work accomplished by the subgrantee Partners and thanked the U.S. Government for the support. The Evaluation Team is satisfied that the field sites were representative of results obtained overall by efforts of the subgrantees. 7.1.5 Level of Community/Beneficiary Participation The Evaluation Team was impressed with the community and beneficiary participation which was assessed to have been as a result of subgrantee community development (mobilization) and community project implementation efforts. In these observed instances, the level of participation was seen as high and more than sufficient to achieve the immediate community objectives. This high level of participation was seen as essential for programmatic sustainability. Enhanced FINAL REPORT 56 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Municipal and Ex Com participation, in some cases higher than Evaluation Team expectations, has the potential to result in increased community project planning and implementation. Participation by IDPs in the community and women was largely appropriate. The Evaluation Team would encourage a higher proportion of IDP involvement in the IDP belt to support increased acceptance in these community settings which may be either temporary, in the currently remote case of resettlement in the near term, or relatively long-term. Participation by women was fairly high. In one case, however, the project was largely influenced by male farmers with interests primarily in animal husbandry, agricultural irrigation and agriculture production In only three settings, Camps 4 and 5 in the Southern IDP belt, the World Vision project in Sumgait, and an IDP neighborhood in Barda, was there a 100% proportion of IDP beneficiaries in the observed AHAP projects. In the rest of the twenty or so sites visited, the proportion of IDP beneficiaries ranged from 10 to 25% in the IDP belt. All project beneficiaries were in conflict-affected areas. No IDPs were observed at the sites visited in Naxcivan as IDPs only constitute approximately 1% of the Naxcivan total population. 7.1.6 Geographic Appropriateness of S.O. 3.1 Activities Community development needs appear nearly universal in Azerbaijan according to the field observations of the Evaluation Team in the areas visited. The geographic appropriateness of Mission activities to follow from the S.O. 3.1 AHAP experience will largely depend upon the extent of resources, both Mission personnel and financial, that can be committed. The Evaluation Team observed ongoing community development needs that could be significantly impacted by the current S.O. 3.1 type activities in the IDP Belt (Central and South), the Sumgait Area, and in Naxcivan. In sum, overall economic conditions, community development, and the nature of USAID￾supported efforts in Naxcivan appear to be approximately two years behind those found by the Evaluation Team elsewhere in Azerbaijan. This appearance is explained, the Evaluation Team believes, by the underlying realities found there. Naxcivan has all of the problems found elsewhere in Azerbaijan. But these have been worsened by two additional constraints. First, Naxcivan is effectively more isolated economically than any other area of the country. With the breaking of the rail link between Naxcivan and "mainland" Azerbaijan, only very high value￾added products justify the expense of air shipment or the tortuous and problematic land shipment through Iran and Turkey. The chief products of Naxcivan are agricultural and few have a value added component that meets this cost threshold. As a result both of the conflict and its subsequent isolation, the entire population of the Autonomous Republic is war-affected. Naxcivan seems in a number of ways simply to be at a lower level of development and growth than the mainland. The only Mercy Corps-supported NGO grantee, ADRA, did not terminate its feeding program until mid-2001 and the need for direct care emergency services has until very recently driven much of its health activities. FINAL REPORT 57 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Community development has been an ADRA focus only for the last two years; unlike the IDP belt, there are no other Mercy Corps Partners in the Autonomous Republic to help support community development through support for projects other than health and microfinance; and ADRA was unsuccessful in its bid to secure a Mercy Corps grant to support cluster and other project activities which have been key to the community development observed elsewhere. If USAID resources permit funding the kinds of community support projects that have proved so beneficial elsewhere, there is every reason to hope that, on a modest scale appropriate to the size of the Autonomous Republic, significant progress is possible. At the very minimum, the Evaluation Team recommends continuing community development, economic opportunity, and health programming in the existing four regions. Of the four regions, Naxcivan was seen as especially impacted by the lack of access to markets and materials. The Evaluation Team recommends that USAID keep Naxcivan as a high priority area for future programming. The engagement in USAID programming by sub-grantees in addition to ADRA would also encourage the synergism observed by the Evaluation Team in the IDP belt in the Central and South programming regions. These include enhanced INGO partnering, support for broader community development infrastructure priorities through CEEOP and SSI-like funding with special attention to potable water, and cluster development. 7.2 HEALTH FINDINGS AND RECOMMENDATIONS Azerbaijan shares with many other parts of the former Soviet Union a heritage of vast hospital over-capacity with more beds than required conjoined with low hospital utilization and excessive lengths of stay in comparison with norms found in Western Europe and the United States. Visits to physicians, clinics and hospitals have been accompanied by an understood requirement for "gratuity" payments. Since independence, in Azerbaijan as in other parts of the NIS, many physicians are unemployed and more have found new employment that make no use of their professional training. While the medical infrastructure – buildings, facilities, equipment – become increasingly outmoded and ever-more deteriorated, the Government of Azerbaijan has been unable to make the investment necessary to restore the former system and unwilling thus far to undertake a conscious top-to￾bottom review and reformulation of its roles, resources, responsibilities and approach. Against this background, there are further declines occasioned by the long war between Armenia and Azerbaijan that began in 1991; a truce was not signed until 1994 and there is still no peace accord. That war led to the displacement of eight hundred thousand to a million persons, the internally displaced persons and refugees (IDPs/refugees). USAID and other donors have sought to patch together effective health care for the country’s populations that are most at risk. Humanitarian assistance by USAID has been undertaken through Strategic Objective 3.1, and largely through an "umbrella" cooperative agreement, first to Save the Children and for the past five years, to Mercy Corps. Through that mechanism, a variety of INGOs have been awarded spacing the USAID-funded Mercy Corp subgrants to provide a wide range of health-related activities in Azerbaijan. FINAL REPORT 58 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 7.2.1 Health Program Effectiveness In virtually all of its current S.O. 3.1 activities, USAID funded subgrantees initiate participatory training to support communities to develop new attitudes directed toward change, empowerment, and the realization that self-help can be a potent force. Individual project accomplishments, in health and other sectors, have been the basis for enhanced community morale, willingness to work together to accomplish further common goals, jointly take on new collective responsibilities, and to see themselves as actors and doers, rather than simply the recipient, for good or ill, of the actions and decisions of others. On this basis, with few exceptions, all of the USAID supported activities in health must be viewed, provisionally, as successes. "Provisionally" because too short a period has passed to judge whether the determination to have some say in their destinies will prove sustainable in the long run. In most of the sites the Evaluation Team visited, the duration of interventions has been only two or three years. Community perceptions of the quality of health care being received and, for health staffs and community users, the value of professional and public health awareness and education training received was without exception seen as substantial. This was especially true with regard to maternal and child health education. Access to potable water at the community taps, made available through several Mercy Corps partners’ projects, was for all an enrichment of great value. Where pharmaceuticals were made locally accessible through revolving drug funds, the value of their accessibility without long travel time and costs was recognized and appreciated In addition, various of the S.O. 3.1 projects represent "new thinking" for the communities served and, even, broader policy discussions and formulation in future. The concepts of revolving drug funds and insurance for health care will be important when, finally, the MOH and others turn seriously to issues of further reforms in the financing of health care, academic training for physicians, primary health care, and system sustainability and reform. The Evaluation Team observed an extensive degree of partnering, cooperation and coordination among and between Mercy Corps’ grantees. Quite simply, no Team member has seen an equivalent or similar level of cooperation and coordination between donor-funded subgrantees. Against these above general conclusions, two exceptions stand out: Pathfinder and AIHA. As noted above, Pathfinder is the only grantee observed whose activities, in their totality, were by intent simply service oriented. Certainly, the beneficiaries with whom the Evaluation Team met expressed substantial appreciation for those services and for those providing them. In several instances observed, caseworkers and beneficiaries seemed to have established close friendships as a result of their interactions over time. Community development, attitudinal change, and a sense of empowerment never arose as spillover benefits. In a country with so strong a history of hospital importance and underutilization concurrent with the practical absence of quality primary health care, the marginal utility of the AIHA activity in terms of needs and resources, seems at best questionable. This is not to say that science-based training of hospital physicians and specialists is irrelevant or unneeded – for surely it is not. The Evaluation Team questions whether, at this juncture, the AIHA approach used here represents the best use of finite US and USAID resources. Large parts of the AIHA award are apparently FINAL REPORT 59 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation expended in travel to centers of grand technology that are achieved through, for Azerbaijan, unimaginably unaffordable costs. The Evaluation Team recommends a far more modest effort in the training of hospital-based physicians, using curricula and techniques that are closely aligned with and reflective of available Azerbaijani technologies and resources, such a training abroad would be more practical, efficient and cost-effective. 7.2.2 Health Issues and Constraints The key issue affecting all health care delivery-related projects is the future role, policy direction, and funding of the Ministry of Health. Continuing growth of the economy, expected governmental revenues from oil, and a willingness to share the results of growth more equitably across the population are important considerations. The Ministry remains the source of approximately half of all health care expenditures in the country, but these are less than two percent of GDP or approximately $7 per capita. Until and unless this rises, or the current financing system, with regard to both sources and amounts, is very substantially modified, there are simply insufficient funds in the system as a whole to secure significant increases in the general population’s health status. Moreover, currently available MOH resources support a highly inefficient delivery system, with too much spent on hospital care and, both relatively and absolutely, too little spent on primary care, disease prevention and health promotion. To the health economist and analyst, it is remarkable – even miraculous – that the apparent improvements in health secured over the last five years have been obtained. The Evaluation Team has been informed by Mercy Corps staff that the MOH has determined that in future, all health training curricula will be subject to its vetting, and that approved training will in the near future be provided only by MOH staff trainers. This is fully consonant with the recommendation of the February 2002 MEDS Report, Social and Health Assessment of Residents, Refugees and Internally Displaced Persons in Azerbaijan. Per se, the Evaluation Team even agrees that, in time and with major reform in the MOH, this is a desirable outcome. But based on observations, The Evaluation Team disagrees that that time is now or, even, in the near future. A major concern is that the general recalcitrance of the MOH, the still-Sovietized style of the medical curriculum and its presentation, the general and apparent absence of openness to new ideas, the paucity of science-based medicine and protocols, and the continuing enthusiasm for hospital-based over primary care will lay a dead hand on practical, open, interactive and needed training if responsibility for it shifts now to the MOH. With regard to the future of USAID support of health care, two key considerations arise. First, USAID’s allocated resources are presently insufficient to meet all the useful and good opportunities to assist that are available in Azerbaijan and choices have to be made. Second, community level health services’ projects are demonstrably a useful wedge through which to initiate broader community activities. They, like other individual projects are seldom, by themselves, enough to secure the broader attitudinal, behavioral and economic spillover necessary to community takeoff. FINAL REPORT 60 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The Evaluation Team, like the several partners, Mercy Corps and USAID sees the incentives posed by the possibility of further financial assistance to follow-on projects as important. Hence the value of multiple partners and programs in the field, and of the cluster approach to spread gains across a broader area. On the other hand, at the humanitarian level, these projects bring individuals and communities a real and immediate benefit: improved health. If, then, resources are narrowly targeted to a small number of communities, synergies exist which may move those communities rapidly forward. With regard to basic humanitarian needs, many will be left behind. Alternatively, spreading limited resources broadly across a larger number of communities, while providing small benefits to many, will not result in creating the critical mass necessary for sustainability. The determination of spread versus concentration lies with USAID. The Evaluation Team concludes that relatively narrow targeting is necessary given resource constraints and the extent of health needs in Azerbaijan. 7.2.3 Health Program Sustainability The judgment of the Evaluation Team regarding sustainability of process, attitudinal and behavioral changes must be provisional. So, too, must be the conclusion regarding financial sustainability of a number of the health-related projects examined. The revolving drug fund sites that were visited seemed to be fully successful, both in Naxcivan and mainland Azerbaijan. The privately operated Fizuli hospital in Babi, built with support to the community by CHF/SII and IMC with initial community mobilization by ISC, is clearly a financial success and its sustainability is assured. The future of other projects, however, is less clear. For example, one of the SII projects visited was a newly built primary healthcare facility, replacing one no longer worth rehabilitating. The building is attractive but virtually empty of the modest equipment needed to render primary care. Assurances received by the Mercy Corps grantee, CHF, prior to construction, that necessary equipment would be supplied by the MOH, have not yet been met although the building has now been completed for some months. The AHAP partners have learned from this experience. IMC has developed a protocol detailing supplies and equipment necessary to the successful operation of such a facility and this protocol will become a part of future projects. As part of the funding of such projects in future, grant funding will be included to provide necessary operational supplies and equipment, as well as for construction of the facility itself. Another example: a birthing center visited in Naxcivan (see the Naxcivan section) and connected to ADRA has in place a revenue-sharing arrangement with the raion hospital. This income should be more than sufficient to assure the birthing center’s financial sustainability. But the revenues expected have been used by the hospital and not returned to the community because the hospital itself is inadequately funded. Several of the partners including IMC, CHF, ADRA offer some form of health insurance, usually on a family subscription basis. The benefit package may vary and several of these also offer fee￾for-service care. IRC offers loans for essential health services beyond the immediate capacity of patients to pay. Take-up rates are highly variable across partners and sites. Health insurance is a new concept to all the communities and the fission that flows from exposure to new ideas is FINAL REPORT 61 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation highly desirable. But whether or not “premiums” can be set high enough to cover all relevant costs and sufficient participation rates can be maintained is as yet by no means clear. The Evaluation Team notes that financial sustainability of many of these projects and their approaches to cost recovery may ultimately depend upon decisions by and about the Ministry of Health. Will private practice be allowed to blossom outside Baku? What will be the role of district head doctors and other officials in sanctioning benefits and cost recovery approaches? Will the MOH work with others and pay its “fair share”? What level of financial support will the MOH receive from government income? The Evaluation Team recognizes that remarkable change opportunities are at hand and a broad array of alternative sustainability approaches are being tested. The outcomes, however, are not yet clear. Nonetheless, the core of the argument for sustainable health projects was well put by IMC’s cost recovery presentation at the October 8-9, 2002 First National Conference on Community Development Best Practices and Future Directions. Included in this presentation were the direct benefits to health as well as mobilization of the community and the institution of transparent accounting practices. 7.2.4 Health Policy and Program Recommendations Health is an appropriate part of S.O. 3.1 and the Evaluation Team recommends that it remain there. At the observed projects sites it was demonstrate that health can be an appropriate fulcrum on which to lever broader community activities and development. It is less clear, however, that its focus should be necessarily limited to the IDP Belt since needs for improved health care seem clear across much of the country. However, the Evaluation Team recommends that there should be a conscious decision to concentrate health and other S.O. 3.1 funds in a limited number of communities to develop over time a critical mass of support for attitude and behavior changes necessary to achieving community and economic development. The IDP Belt, urban concentrations of conflict-affected persons, and Naxcivan appear to be the areas of greatest need. The Evaluation Team, along with USAID/Azerbaijan, recognizes the difficulties associated with working with the Ministry of Health. Nonetheless the conclusion cannot be escaped that in health, sustainability, indeed, even viability, is dependent upon MOH decisions and that those decisions at least may be better if outside assistance can be brought to bear. Moreover, in their day-to-day activities, the INGO partners of Mercy Corps not only must deal with ExCom officials, municipality leaders, but also with district hospital head doctors and Ministry of Health officials. Three events will soon occur, which may provide USAID with opportunities to deal from a relatively protected position with the Ministry of Health. A contract will soon be let, on behalf of the Ministry of Economics, to study health care expenditures in Azerbaijan. USAID has supported similar work in a number of NIS countries. USAID could indicate its interest in at least observing, and perhaps at a very modest level, participating in such a study. A possible point of focus, perhaps, is the considerable volume of health activities supported by USAID in both the IDP Belt and Naxcivan. FINAL REPORT 62 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The second event is the expected release of the CDC reproductive health study very late this year. The interim report enjoys the imprimatur of both USAID and the MOH. The final report is expected to show maternal and infant morbidity and mortality outcomes far worse than those of the MOH. If resources are available, “packaging” the CDC report with an offer of modest assistance may help address these problems. Third, the World Bank’s health reform project is beginning in Azerbaijan, with a district hospital in Naxcivan, where ADRA, supported by USAID through Mercy Corps, has quite good relations with the Republic’s government, one of the five sites selected for a pilot project. Usually, these loans include some policy development component – again an activity with which USAID has been associated in a number of NIS and other countries. This, too, may be an opportunity for USAID interjection and involvement. 7.3 ECONOMIC OPPORTUNITY The Partner organizations under a total of seven subgrants are engaged or have engaged in economic opportunity programming under the AHAP umbrella. ACDI/VOCA conducted a program of Community Economic Recovery Groups (CERGs) that has been extended, removed from the AHAP umbrella, and placed into S.O. 1.3, Economic Development to be managed directly by USAID/Azerbaijan personnel. CHF, under a single subgrant, implemented the Social Investment Initiative (SII) in three of the four USAID conflict-affected areas and is the sole subgrant to be conducted in more than one impacted area. Save the Children and CHF are the Partner organizations with two separate subgrants for economic opportunity. ACDI/VOCA conducted a program of Community Economic Recovery Groups (CERGs) that has been extended and removed from the AHAP umbrella. The three remaining AHAP Partners, International Medical Corps (IMC), Pathfinder, and World Vision are not directly involved in economic opportunity. IMC is conducting cost recovery components for health services which do have economic ramifications. 7.3.1 Findings and Conclusions The EO sites visited by the Evaluation Team did show an impressive level of success. A contrast of impact was clearly demonstrated at a wool cleaning project when the ditch that had previously been used was pointed out by project beneficiaries. It was contended that working in that ditch to clean wool resulted in illness and constituted a hardship to achieve wool cleaning results. The Evaluation Team subsequently observed women in the Bilasuvar Camps who were using ditches for washing raw wool as well as for washing clothing and concluded that this was a common practice in the conflict-affected areas. Loan projects conducted by SC in the Central region of the IDP belt and by ADRA in Naxcivan appeared to have supported borrower enterprises with relatively few failures (an estimated failure rate of 10% by the Evaluation Team). Repayment rates were at approximately 98%. The conclusion from Evaluation Team visits to these Partner selected sites is that the loan projects were working as planned. FINAL REPORT 63 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Three focus groups, composed of persons with no or very little contact with AHAP programming, were conducted. Two were in the Central region and one was in Naxcivan. Individuals in these groups, in addition to identifying community needs, stressed the lack of economic activity in their areas. The communities visited in the urban area near Baku were noticeably in need of additional economic activity. The obvious conclusion from these focus group sessions and by observation in the urban target area is that economic opportunity projects are widely needed. 7.3.2 Recommendations The economic opportunity projects, conducted under CHF SII, supported community groups in designing and implementing several projects of community-wide interest and impact. These projects were in irrigation and agricultural product processing. Removing these kinds of economic opportunity projects would weaken the community mobilization and community development efforts that have been shown to have an enhanced effect through addressing economic opportunity concerns. The Evaluation Team recommends that such community-wide economic opportunity projects remain within the revised S.O. 3.1 programming. The microfinance projects of ADRA and Save the Children, on the other hand, have not demonstrated that they contribute significant synergism in support of either community development or health programming. Such projects and grants could be moved to direct USAID management under S.O. 1.3. In making that move, additional regional responsibility will result for S.O. 1.3 since these microfinance projects are currently in Naxcivan and the Central Region of the IDP Belt. 7.4 COMMUNITY DEVELOPMENT FINDINGS AND RECOMMENDATIONS The Evaluation Team found that all of the subgrantee programs affected the communities in which they operated in one or more ways. Of the Partner organizations, IRC appeared to have the strongest and most coherent program of community mobilization and development. IRC utilizes a proven participatory training process that has been improved and applied in a variety of other IRC country settings. The IRC community mobilization begins with a four day intensive program for up to 150 members of a community or collection of communities. Subsequent training is provided and support is given for microprojects that in subsequent cycles can be as large as $5,000. Several of these community groups were supported in realizing their community priorities through linkages with CHF. The CHF program provided grants in the $20,000 range that allowed these communities to develop substantive and quality community projects. 7.4.1 Inclusion of Economic Opportunity Projects The Evaluation Team recommends the inclusion of economic opportunity as important to Community Development in those instances where the majority of the community population directly benefits. An example would be an irrigation project where the community consists primarily of farmers who are all directly affected by such a project. Accordingly, in current and future Community Development programming, these cooperative-type community efforts should FINAL REPORT 64 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation be permitted under the technical assistance and funding that is provided by the Partner organizations that are engaged in Community Development. 7.4.2 Levels of Community Project Funding Larger funding for community projects available through CHF was observed to have resulted in quality efforts, an increase in community pride, and a strong sense of community achievement. The Evaluation Team recommends that future community programming be structured to allow community project funding in the $15,000 to $25,000 range. Where gradual phasing of the amounts made available to community groups is used as a training mechanism, the smallest grants should still be sufficient to impact on the typical projects that have historically been undertaken in that region. The Evaluation Team recommends subgrant Partners consider the awarding of community project grants at minimums large enough to assure that in a community structure such as a school that the roof will provide a sufficient cover from the elements. 7.4.3 Community Mobilization/Participatory Methodologies Other Partner organization project efforts in Azerbaijan could benefit from the application of the community mobilization/development methodology of IRC. While participatory methodologies are applied to one degree or another, the structured approach of IRC was seen to result in strong community groups that accepted the responsibility for planning and taking actions to improve their communities with or without international donor support. Achievement of such a level of self-actualization/realization does not occur without a substantial commitment to support such a community breakthrough. The Evaluation Team recommends that current and future community development efforts require Partner staff training to the extent that similar support can be provided to other community groups in the implementation of the various AHAP approaches to community development, health, and economic opportunity projects. 7.5 AHAP UMBRELLA PROGRAM FINDINGS AND CONCLUSIONS The MCI operation of the AHAP umbrella meets or exceeds the mandate to Mercy Corps set forth by USAID in the original Cooperative Agreement. Every grant-funded project, regardless of its sectoral focus, apparently has as its center a community organization element that seeks to build sustainable capacity in target communities. This approach responds to IR 3.1.1.2- Communities Organized to Address Self-Defined Needs. It is also one of the key ways of achieving IR 3.1.1-Vulnerable Communities Better Able to Meet Their Own Needs. The substantive content of grantee activities, mostly health and economic development, including micro-finance programs, as well as school and road construction, responds to achieving IR 3.1.1.1-Increased Access to Economic Opportunities and Support Services and IR 3.1.1.3 Communities Have Access to Better Quality Services. Likewise, Mercy Corps’ approach under AHAP has effectively moved the umbrella from direct service to developmental programming. The AHAP projects under the umbrella are now providing no direct humanitarian assistance. Most are located in the "IDP belt" and include IDPs as participants with – in most instances – some IDP membership on community committees. FINAL REPORT 65 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 7.5.1 Management Structure of the Umbrella Mechanism The AHAP umbrella has been extremely effective in quickly mobilizing subgrantee Partners and implementing appropriate programming. AHAP I RFAs were issued in March 1998, one month after Mercy Corps was awarded the agreement to manage the program. The second round of RFAs was issued for AHAP II in a similarly timely fashion. The selection process itself appears to be fair, transparent and efficient. The process, modeled on USAID’s own procurement procedures, seems rigorous and objective and appears to have operated more quickly than similar large scale USAID procurements. Respondents are given approximately six weeks from RFA issuance until the deadline for proposal submission. The RFAs themselves are clear and complete, providing specific evaluation criteria and eligibility requirements, appropriate background data, including the appropriate sector strategy prepared by Mercy Corps in its own proposal submission to USAID. 7.5.2 Umbrella Effectiveness and Innovations The most striking and most valuable aspect of the AHAP umbrella—and of Mercy Corps’ management of it—is the atmosphere of collaboration and cooperation that exists among the Partner organizations. It rarely is found to such a degree and greatly increases the impact and effectiveness of the AHAP Partnership. This collaboration was demonstrated in many of the AHAP project sites in the IDP Belt where two and as many as three Partners were engaged in the same community to develop and implement complementary programs. This level of integration of partner activities is an innovation of the Mercy Corps umbrella. The Mercy Corps umbrella has been successful in bringing a wide range of experienced and committed PVOs to address the program areas of S.O. 3.1. The original approach that Mercy Corps has taken is that, from the outset, of monitoring was on a geographic basis rather than grantee by grantee. This may have contributed to the results achieved by the overall umbrella program. 7.5.3 Umbrella Issues AHAP technical assistance has not always been effective in practice and is viewed by some as redundant in light of the experience and technical expertise of grantees, nor is it a part of the original USAID mandate to Mercy Corps. The AHAP Partners are among the most experienced and respected in their fields. The Evaluation Team observed that partner organizations improved their field projects according to real project situations and opportunities. Accordingly, some of Mercy Corps’ resources now committed to monitoring and technical assistance could be redeployed to coordination with USAID which would have greater payoff in terms of programmatic quality. The quality and reliability of monitoring data appear to be good; its meaningfulness and value in applied forward decision-making is open to question. The Evaluation Team found no Partner personnel who had knowledge of program analyses that were derived from the monitoring FINAL REPORT 66 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation information. The process of integrated monitoring apparently did support Partner coordination and cooperation. The area in which Mercy Corps has been least successful is creation of trust between grantees and itself. The fact that Mercy Corps is so central to grant award decision-making raises some serious questions of conflict-of-interest. 7.5.4 S.O. 3.1 Strategy Development and Implementation According to Evaluation Team interviews, Mercy Corps has had wide latitude in determining the particulars of how the USAID’s S.O. 3.1 Strategy is developed and implemented. The involvement by USAID/Azerbaijan personnel has primarily consisted of review and approvals of RFAs and subgrant awards. The relatively small role of USAID/Azerbaijan in designing and managing the evolving program strategy seems a missed opportunity, both from the standpoint of fine-tuning its work in S.O. 3.1 and lessons that would inform its larger assistance program in Azerbaijan. Without damaging the management efficiencies inherent in the umbrella, USAID could profit from more active involvement in determining program direction and evolution. Despite the copious collection of information and assiduous reporting, the evolving program direction emerges more organically from Mercy Corps management and experience than from a conscious effort to analyze and interpret the monitoring data. This is not to say that there is anything wrong with program direction. It already has been established that Mercy Corps has mounted an excellent program, relevant to USAID’s Strategic Objectives and the needs of target communities. Rather it suggests that program evolution is more dependent on Mercy Corps’ institutional memory and expertise—both in Azerbaijan and at headquarters—than on the use of data. 7.5.5 MCI Monitoring and Compliance The monitoring function is the central vehicle for Mercy Corps’ approach to managing the AHAP umbrella. It is combined with a technical assistance function, which it should be noted is not a part of the USAID mandate to Mercy Corps. Mercy Corps has utilized an integrated approach in AHAP to bring the full panoply of services in economic opportunity, health and community development to bear on target communities. Reflecting this approach, it has adopted an integrated approach to monitoring, in which a team of Mercy Corps personnel visits grantees for up to five days. In addition to the monitoring visits, each grantee provides reports to Mercy Corps based on some 150 indicators. A number of grantees indicated that Mercy Corps monitoring is burdensome because of its frequency—recently reduced to quarterly from every two months—combined with the depth and duration of visits as well as required reports. They pointed to time consumed not only by the visits themselves but the advance preparation required of them. The Evaluation Team also recommends that the number and type of program indicators be reviewed and reduced. On balance it seems that Mercy Corps’ monitoring is excessive and that the frequency should be reduced to semi-annual. FINAL REPORT 67 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation In general, the compliance function of assuring subgrantees meet USAID policies, procedures and regulations is carried out well. Few grantees indicated concerns about lack of information or misinformation nor did USAID indicate major concerns about Mercy Corps’ ability to secure satisfactory compliance. 7.5.6 Umbrella Mechanism as an Implementing Vehicle To date, the umbrella has been an effective management tool for USAID, which has assigned one individual as the internal management capacity. The umbrella has yielded a high quality and relevant program. It has had the subsidiary benefit of allowing for a broad mix of organizations to operate in Azerbaijan through its effective process of subgrantee RFAs, more so than may have been possible had USAID been managing all of the S.O. 3.1 activities directly. In addition, and very importantly, the umbrella has fostered a unique level of coordination. Mercy Corps has managed the umbrella very efficiently. Of a total six-year budget of $45 million it is anticipated that more than $37 million will have been awarded in grants, making the annual management cost approximately $1.3 million. Mercy Corps has a low indirect cost rate of 17 percent and is very frugal in its home office management charges. In addition, its salary structures and especially its expatriate allowance structures are significantly lower than those of USAID. Unquestionably, USAID has received good value from Mercy Corps’ management of the AHAP umbrella. Further, the Evaluation Team recommends a narrower mandate for any future umbrella, particularly in limiting the technical assistance function and reducing the monitoring function, to result in fewer expatriates on the umbrella payroll, further diminishing management costs. It is doubtful that USAID’s costs would be lower in managing the umbrella directly. To do so, especially with the numbers of implementing organizations involved, would require at least one additional expatriate staff person in addition to the Humanitarian Assistance Officer now on board and several additional local staff persons. It also would require two or three additional financial management personnel and attendant costs of increased transportation and office space. There may be internal issues of Mission operations versus program budgets. The real tradeoffs involve programmatic diversity and to some extent quality. Using the $1.3 million it now annually spends on Mercy Corps management, USAID could mount a creditable program. Direct management by the Mission, however, may require limiting the number of program implementers in the interest of management efficiency. It also would sacrifice the intensive administrative and financial oversight AHAP has so far enjoyed as well as a large measure of the interagency collaboration—and the resulting synergies—the umbrella has provided. The choice for USAID on using an umbrella manager or adding the necessary technical, administrative, and contracting personnel to Mission staff will be a function of internal priorities. Where do the activities now carried out under S.O. 3.1 fit in the larger scheme of USAID objectives? Do they support other, more important objectives or are they overriding? If the FINAL REPORT 68 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation current S.O. 3.1 objectives—especially community development—are not central features of the USAID strategy, the intensive level of oversight provided by the umbrella may be redundant. What are the feasibility and desirability of adding USAID staff? Since additional staff could be deployed in a variety of areas, are they best used on activities now carried out under S.O. 3.1? 7.6 UMBRELLA DEVELOPMENT/IMPLEMENTATION RECOMMENDATIONS All things being equal, a continued umbrella is preferable to direct management by USAID. The major reasons are the more intensive oversight and monitoring that an umbrella can provide and the richer mix of implementing organizations it affords. The ultimate decision of USAID/Azerbaijan will of course depend on considerations about its overall assistance program in Azerbaijan, available resources and internal management capacity. On balance, a PVO umbrella manager is the logical manager for a future umbrella, both for reasons of cost and of culture. The shared cultures of the PVO Mercy Corps and its PVO subgrantees have yielded positive results that a for-profit umbrella manager would be unlikely to duplicate easily. USAID/Azerbaijan and its S.O. 3.1 programs will benefit from development of a new USAID strategy that reflects the current development environment in Azerbaijan. The PMP that is developed for the new strategy should focus on qualitative information relevant to the fundamental objectives the S.O. 3.1 program is aimed at achieving and should have many fewer indicators than the 150 output-related indicators in the current plan. 7.6.1 Umbrella Review and Program Strategy If the umbrella mechanism is continued, USAID/Azerbaijan should institute a system of formal review of program progress with the umbrella manager. The review would address issues such as achievements to date, possible refinements, lessons learned and future planning. The review should be conducted semi-annually, using the umbrella manager’s report as its basis. USAID should take whatever legal measures are available to it to discourage the umbrella manager from undertaking implementation activities in Azerbaijan while it holds the umbrella agreement. This would include non-compete language in any future RFAs for umbrella management. The umbrella manager should spell out the means by which it would minimize both the appearance and reality of conflict-of-interest arising from the knowledge of proprietary information at its disposal arising from its management of grants to other U.S. PVOs in the umbrella partnership. The umbrella manager should remove itself entirely from grant award selection and use disinterested experts for review of proposals and recommending to USAID. This does not mean that the umbrella manager would not participate in the review process in an advisory capacity that took advantage of its knowledge of the program and its requirements and of strategic FINAL REPORT 69 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation planning objectives for the program. USAID will, of course, retain final approval authority for subgrants. 7.6.2 Staffing and Personnel Resources In a future umbrella, staffing should be reduced by approximately 10 persons. The umbrella itself should have no more than three expatriates (Chief of Party, Program Director and Chief Financial Officer). De-emphasis on internal technical assistance and reduced monitoring frequency would enable these reductions. To fill the potential gap occasioned by a reduction in full-time internal technical assistance capacity, the umbrella manager should budget funds for short-term technical assistance, as required by technical requirements of umbrella administration. The umbrella manager should maintain a database of appropriate consultants with specialized areas of expertise should grantees require assistance in locating the required skills. Similarly, grantees should be encouraged to budget for consulting in their respective budgets. 7.6.3 Monitoring and Integration The umbrella manager should employ the highly successful integrated monitoring system developed by Mercy Corps. Monitoring frequency should, however, be decreased to semi￾annual. Mercy Corps’ monitoring has been a very effective tool in identifying programmatic weaknesses and needs for outside expertise; the umbrella manager should continue to make this aspect of monitoring a priority and should draw on the technical assistance clearinghouse to meet needs it identified. 7.7 MISSION STRATEGY DEVELOPMENT The Evaluation Team recommends that USAID/Azerbaijan consider the development of a structure of strategic objectives that recognizes the accomplishment of an evolving program in the conflict-affected areas from basic humanitarian needs of food and shelter that are being provided by other donors and, to some extent, by the GOAZ. While there is substantial need for economic opportunity programming, the general economy, even in Naxcivan, has been improving over recent years as shown in World Bank economic indicators for Azerbaijan over the past five years. Gross domestic product has increased at high rates and the inflation rate has been substantially decreased. USAID/Azerbaijan has already initiated the process of defining strategic objectives to develop a strategy statement that reflects the improving humanitarian situation in Azerbaijan and the impact of previous and current USAID programming. It is recommended that this strategy statement continue to have a humanitarian oriented context. It is recommended that since basic human needs have been met at some level, the revised S.O. reflect the phasing to broad-based community and economic development to targeted communities of greater need and potential as contrasted with S.O. 1.3 target areas of greater opportunity. The required set of RFAs, RFPs and extensions as appropriate should be completed over the next seven to nine months to allow an FINAL REPORT 70 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation orderly program development and minimum disruption of correlated USAID current projects by January, 2004. The revised strategy should also take advantage of programmatic insights that may be developed from the Cluster Development subgrant extensions of World Vision and IRC. In addition, IRC will be implementing a Municipal Government oriented subgrant extension that may provide relevant findings for S.O. 3.1 strategy development. The Evaluation Team also recommends that the revised strategy consider approaches to strengthen the local NGO community. 7.7.1 Government Interaction There is a much-improved climate for interaction with the GOAJ since the presidential waiver of Section 907 of the FREEDOM Support Act. It now can extend to financial assistance, where appropriate, but more important are the opportunities for direct engagement with officials and advocacy on concrete issues that affect the well being of the S.O. 3.1 target population. To be sure, in the field there has been contact all along between implementers and local and regional level officials since it would have been impossible to engage in assistance activities, particularly of a transitional nature, without their knowledge and at least tacit approval. But the contact has been narrowly limited since implementers were aware of the consequences to their program if they were seen to be operating in violation of Section 907. Nonetheless, there are many examples of activities in which governmental authorities have seen the value of the community-level work being carried out under AHAP and have of their own volition offered modest financial support, office space, or the use of land. This occurred across the sectoral spectrum but was most evident where community projects involved facilities like schools and health clinics that are under the jurisdiction of local and regional government. Equally there are numerous examples where there was no government response to community initiatives, as in the case of a refurbished clinic in the Southern region of the IDP Belt, that was without equipment of any kind, or at a nearby village, where after initially encouraging clinic repair, the local doctor withdrew his support. Both circumstances suggest the value now of systematic engagement with the government on specific issues. Issues can be as simple as equipping refurbished clinics and schools or more complex, such as acknowledging and perhaps even encouraging official adoption of cost recovery and health insurance mechanisms. Some levels of engagement would be little more than educational, trying for instance to make government more aware of the importance of a uniform policy on microfinance interest. This might in the future avoid the unfortunate circumstance of the difference between sustainable interest rates of various AHAP partners and the unaccountable decision by The World Bank to offer below-market rates for its government-sanctioned loan program. Several informants made the case for concerted advocacy on such matters. One of the most articulate was IRC’s outgoing Community Health Sector Coordinator, who argued that issues need to be tackled in a coordinated fashion, beginning at the local level and moving upward to FINAL REPORT 71 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation the national level. This is now possible and desirable, especially inasmuch as partners at the local level, AHAP at the regional level, and USAID at the highest echelons provide an excellent framework for coordinated advocacy. Such an effort no doubt would be enhanced by modest financial incentives in the form of financial assistance to government to underwrite its work in the areas concerned. Fortunately, this now is possible. The Evaluation Team makes these suggestions in the full knowledge of the difficulties involved in dealing with a highly autocratic and centralized government system which, in part because of 907, has not been accustomed to close collaboration, especially with U.S. development assistance. For these reasons, focused advocacy efforts on issues directly beneficial to S.O. 3.1 beneficiaries should begin slowly and USAID should content itself with small gains, especially in the beginning. 7.7.2 Sustainability The sustainability of S.O. 3.1 activities is uncertain. Most of projects that the Evaluation Team saw were new, often less than one year old. Thus, it is not clear that the initial community enthusiasm that resulted in economic development projects, schools and clinics, will be maintained over the long term. Nor is it clear that community members, after brief exposure to the ethic and potential benefits of community mobilization have the skills to move forward consistently. SII is in this sense a very important AHAP initiative because it provides the opportunity for particularly promising communities to scale up to more ambitious projects and equally important to reinforce their experience and hone their skills. Similarly, the cluster approach is a positive step in the direction of building on past successes and creating critical mass as well as reinforcing community cohesion. Future S.O. 3.1 programming should seek to consolidate this approach. Close collaboration with local NGOs is the usual strategy for ensuring that the interventions of external organizations can build a local capacity for carrying them on and expanding them. In the NGO enabling environment that now pertains in Azerbaijan, where the GOAJ is threatened by local NGOs and is reluctant to recognize or register them, this has not been a viable strategy. AHAP has done the “next best thing” by creating numerous opportunities for local staff of partner organizations to build their skills and technical knowledge. They undoubtedly will represent a potent force for change over the long term in Azerbaijan. Now, however, there is the opportunity to go further, in an admittedly hostile environment, to address the enabling environment itself. The Evaluation Team is aware, from its conversations with USAID Democracy and Governance personnel, how difficult this will be, especially at first. But it seems important to begin the process of strengthening local capacity. A first step would be, perhaps, to engage an organization like the International Center for Not-for-Profit Law (ICNL) to help devise a strategy for beginning a dialogue with the GOAJ on a more enlightened NGO enabling environment. FINAL REPORT 72 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation This is of course more the purview of USAID’s Democracy and Governance unit but it should be carried out in close collaboration with S.O. 3.1 implementers. To the degree that concrete steps evolve and are effective, future S.O. 3.1 programming should be adapted to complement it and further its aims. 7.7.3 Monitoring and Follow up Monitoring of completed projects has been left to individual subgrantee implementers and there has been no formal mechanism for including this important data in forward planning. Moreover, such post-project monitoring has been spotty and sometimes not carried out at all because projects have ended or other imperatives have intervened. The Evaluation Team recommends that consistent post-project monitoring should be the specific and explicit responsibility of S.O. 3.1 managers, whether USAID or an umbrella manager. At the very minimum, spot checks, designed to measure independent activity after the end of formal engagement with implementers and the continued cohesiveness of community groups, should be made at 3, 6 and 12 month intervals. The data so derived would be an important part of determining the nature of future activities, their locations, the need for follow-up training and especially the composition of cluster activities. FINAL REPORT 77 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ANNEX A DRAFT SELECTED BIBLIOGRAPHY ADRA/Azerbaijan, Nakhchivan health development program, undated. ADRA/Azerbaijan, Nakhchivan Enterprise Development program, July, 2002. Mercy Corps Agenda, National Community Development Conference, Baku. Oct, 2002. Bob Leonard, Community Development Approaches in Azerbaijan for the First National Community Development Conference, Oct 8-9, 2002. CHF- Azerbaijan. Social Inventory Assessment, 2001. David H. Holt, David T. King, Edward Hoffman, David Harmon, Private Sector Assessment: Azerbaijan, USAID/Azerbaijan, Nathan – MSI Consortium Arlington, VA, November 5, 1999. Dennis M. Smyth, Elizabeth Osborn, USAID/Caucasus Azerbaijan. Performance Monitoring Plan for Strategic Objectives 1.3, 2.1, 3.1 Sep,2000. Florina Serbanescu, Leo Morris, Shafaq Rahimova, Lisa Flowers, Women’s Reproductive Health Survey, Azerbaijan 2001, Division for Reproductive Health, Center for Disease Control and Prevention, US Department of Health and Human Services, (Atlanta, GA), April 2002. George Gotszade, USAID Technical Report, Primary Health Care Network Survey for Southern Azerbaijan, International Medical Corps. Community Based Primary health Care Development Program. November, 2000. George Gotszade, USAID Technical Report. Population Health Needs and Health Service Utilization in Southern Azerbaijan, International Medical Corps Community Based Primary Health Care Development Program, November, 2000. (Partners: Pathfinder, World Vision, CHF, World Vision/Community Development). Implementing Partner Success Stories, Urban region. International Medical Corps. IMC Framework for Decision making. Health Infrastructure Micro￾Projects, 2002. Journal of the American Health Alliance, Common Health, Spring, 2001. Mercy Corps, Revised Proposal to the USAID, Azerbaijan Humanitarian Assistance Program, 1998-2002, February 7, 2000. FINAL REPORT 78 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Mercy Corps, Azerbaijan Humanitarian Assistance Program, Semi Annual Report #7 for Period. December 1, 2000 to May 31, 2001, August 01, 2001. Mercy Corps/Azerbaijan AHAP revised budget. June 15, 2001 Mercy Corps/Azerbaijan Annual Monitoring Plans for AHAP Grantees. Mercy Corps/Azerbaijan AHAP revised budget. June 15, 2001 Mercy Corps/Azerbaijan, Azerbaijan Humanitarian Assistance Program Bulletin, August 2002. Mercy Corps/Azerbaijan, Economic Opportunities Bulletin, September 2002. Mercy Corps/Azerbaijan, RFA for Community Development Program, February 14, 2000. Mercy Corps/Azerbaijan, RFA for Economic Opportunities program, February 14, 2000. Mercy Corps/Azerbaijan, RFA for Social Investment Initiative, Oct 6, 2000. Mercy Corps/Azerbaijan RFA for Integrated Community Development Program, March 2, 2001. National Enviroment Secretariat, Government of Kenya, et al. Participatory Rural Appraisal Handbook, Worcester: Clark University, 1994 Tim A. Clary, Manucka Djibuti, Barry Silverman, Ronald Wilson, USAID, Social and Health Assessment or Residents, Refugees and Internally Displaced Persons in Azerbaijan, LTG Associates, Inc & TvT Associates, Inc. February, 2002. USAID/ENI/DGSR, Technical Assistance Project, Final Report, Caucasus Health Sector Assessment, Oct 1998. USAID/Caucasus, Azerbaijan Strategic Plan 2001-2003. June 2000. USAID, Cooperative Agreements to Mercy Coprs, EE-A-00-98-00005-00, January 15, 1998. January 14, 2001. USAID, Cooperative Agreement Modification 1-11. USAID/Azerbaijan, Complete List of USAID partners, October 4, 2002. USAID-Mercy Corps, Management & Communication Protocol, undated. U.S. Department of State, Azerbaijan, Country Report on Human Rights Practices, 2001, The Bureau of Democracy, Human Rights & Labor, March 4, 2002. Websites: FINAL REPORT 79 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation USAID/Washington, Building Democracy in Azerbaijan. http://www.usaid.gov/democracy/ee/azerbaijan.html. 9/26/2002 USAID/Azerbaijan, Program Data Sheet, http://www.usaid.gov/country/ee/az/112-0210.html USAID/Azerbaijan, Regional Overview, http://www.usaid.gov/country/ee/az/112-0310.html 9/26/2002 USAID: Azerbaijan Environment Resources Links, http://www.usaid.gov/regions/europe_eurasia/countries/az/azerbaij.htm 9/26/2002 USAID, Azerbaijan Regional Overview, http://www.usaid.gov/pubs/cbj2002/ee/az 9/26/2002 FINAL REPORT 80 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ANNEX B INTERVIEW/MEETING/FIELD VISIT SCHEDULE October 7, 2002-October 25, 2002 Monday, October 7, 2002 Time Activity Lead / Interview 9:00--10:00 Grantee visit: American International Health Alliance William McKinney, Country Coordinator, USAID/ Azerbaijan 10:00--10:15 Meet USAID Staff Gulnara Rahimova, Program Development Specialist, USAID 10:15--11:15 SO 3.1 Humanitarian Response Overview Valerie Ibaan, Humanitarian Response Program Specialist,USAID 11:15--12:15 Discussion of Schedule Gulnara Rahimova, Program Development Specialist, USAID 12:15--13:00 SO 3.1 Program History/Background Khalid Khan, USAID/Georgia former Humanitarian Response Program Specialist; and Valerie Ibaan, Humanitarian Response Program Specialist, USAID 13:00--14:00 SO 1.4 Business Development Overview Working Lunch--Embassy Cafeteria Peter Duffy, Private Sector Advisor, USAID 14:00--15:00 SO 2.1 Democracy Overview Elgin Guliyev, Project Management Specialist, USAID 15:00--18:00 Mercy Corps International Visit Bill Holbrook, MCI Team Leader & Staff; and Steve Zimmerman, Chief Operatoring Officer, Mercy Corps International HQ Tuesday, October 8, 2002 Time Activity Lead / Etc. 8:00--8:30 Draft Evaluation Work Plan Valerie Ibaan, Humanitarian Response Program Specialist, USAID 8:30--12:00 Opt 1: Attend Community Development Conference Gulnara Rahimova, Program Development Specialist,USAID 12:00--14:00 Travel and Lunch with Khalid Khan for SO 3.1 Presentation/ Background Khalid Khan, USAID/Georgia former Humanitarian Response Program Specialist 14:00--15:45 Interviews for Interpreters and Drivers 15:45--17:30 Grantee Visit: Community Habitat Foundation (CHF) Beverly Hoover, Country Director,CHF FINAL REPORT 81 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Wednesday, October 9, 2002 Time Activity Lead 9:10--10:40 Grantee Visit: American International Health Alliance Jeyhun Mamedov, Program Coordinator 11:00--12:00 Meeting with the Ambassador at Embassy William Mckinney, USAID/Azerbaijan; and U.S Ambassador Ross Wilson 12:15--13:45 Grantee Visit: International Medical Corps (IMC) Adam Sirois, Country Director, IMC, Fuad Hamidzade, IMC 13:45--15:00 Lunch 15:00--16:45 Grantee Visit: International Rescue Committee (IRC) Pamela Husain, Country Director, IRC 16:45--18:15 Grantee Visit: Pathfinder International Ramin Hajiyev, Country Director, Pathfinder International Thursday, October 10, 2002 Time Activity Lead 9:00--10:30 Grantee Visit: Save the Children Tryggve Nelke, Field Office Director and Staff 10:45--12:15 Grantee Visit: Adventist Development & Relief Agency (ADRA) Wagner Kuhn, Country Director, ADRA 12:15--13:15 Lunch Afternoon World Vision Field Visit: Sumgait 45 minute transit Michael McIntyre Pathfinder International Field Visit: Sumgait Ramin Haiyev, Country Director, Pathfinder International Friday, October 11, 2002 Time Activity Lead 9:30--11:30 Visit Sabirabad Qaratapa 11:30--11:45 Travel to Sairabad Ahmedabad 11:45--13:00 Visit Ahmedabad 13:15--14:15 Lunch 14:15--14:45 Travel to Sabirabad Kurkand 14:45--15:30 Visit Kurkand 15:30--16:00 Travel to Sattli Varkankand Box Car Camp 16:00--17:30 Visit Varkankand 17:30--19:30 Return to Baku via Sattli Town Railway Station Tuesday, October 15, 2002 Time Activity Lead 9:00--9:30 Partners Meeting 9:45--10:30 Mughanly (Barda) Barat Azizov, SC Community Mobilization Officer FINAL REPORT 82 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 10:50--11:35 Garadaghli (Barda) Seymour Yusifli, Senior Project Officer 12:05--12:50 Eravanli (Ter-ter) 13:00-14:20 Lunch 14:30--15:15 Yarmarka (Barda) 16:05--16:50 Gulovsha, Yeulakh 16:50--17:40 Travel to Barda Wednesday, October 16, 2002 Time Activity Lead 10:00--10:30 Visit Development Resource Center 10:30--12:00 Focus Group Discussion in DRC 12:00--12:30 Travel to Fizuli Ahmedbayle 12:30--13:30 Lunch 13:30--13:45 Travel to Fizuli Babi 13:45--15:30 Visit Babi, Qayri 15:30--16:15 Travel to Beylagan Garadagli 16:15--17:15 Visit Garadagli, Fialjtovka 17:15--18:00 Return to Imishli Thursday, October 17, 2002 Time Activity Lead 9:30--10:30 Staff Meeting 10:30--11:00 Visit Imishli Hospital 11:00--11:30 Travel to Bilasuvar Camp 4 11:30--13:00 Meet Community Group Members 13:00--14:00 Travel to Bilasuvar Town 14:00--15:00 Lunch 15:00--17:00 Various Site Visits Monday, October 21, 2002 Time Activity Lead 9:45--12:00 Meeting with ADRA Staff Jenny Sequeria and Mark Castellino 12:00--13:00 Meeting with Deputy Prime Minister Tarana Guliyeva, Deputy Prime Minister 13:00--13:30 Lunch 13:45--19:00 Group 1 to Arazin, Julfa Region (Micro-credit Group); Aza, Ordubad Region (village Health Council Elchin Najafov, ADRA; Karim Guliyev, ADRA;Matanat Babyeva, ADRA; Group 2 Arafsa, Julfa Region (both Micro-credit and Village Health Council) FINAL REPORT 83 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Tuesday, October 22, 2002 Time Activity Lead 9:45--12:30 Focus Group organized by Government Group 1 to Mamukend, Sharur Region (to observe ADRA/MOH EPI Training), Ashagi Yayji, Sharur Village Health Council Meeting Abulfaz Talbov, ADRA; Mohamma Jabbarov, ADRA; Nadir Ahmadov, ADRA; and Tarana Guliyeva, Deputy Prime Minster Group 2 leave for Kerimbayli, Zenadin (Babek Region) and other nearby villages to meet with credit groups 12:30--1:30 Lunch 1:30--2:30 Meeting with Speaker of Parliament for Naxcivan Autonomous Republic Mr. Vasif Talibov, Speaker of Parliament for Naxcivan Autonomous Republic Wednesday, October 23, 2002 Time Activity Lead Meet with Cabinet of Ministers Ali Hasanov; Deputy Prim Minster; Head of the State Committee for Refugees and IDPs; Head of the Republican Commission of International Humanitarian Assistance; and Gruban Sadigov, Assistant to Head of the Republican Commission of International Humanitarian Assistance Presentation Outline of Evaluation Report and Interim Report at USAID office William McKinney, USAID/Azerbaijan; Valerie Ibaan, Humanitarian Response Program Specialist, USAID; and Kent Larson Mercy Corps Office Visit William Holbrook, Team Leader, MCI Thursday, October 24, 2002 Time Activity Lead Mercy Corps Vafa Asadova, Community Development Program Officer; Sue Leonard, CD/SII Programs Manager; and Ziba Guliyeva, Social Investment Initiative Program Officer Friday, October 25, 2002 Time Activity Lead USAID Kelley D. Strickland, Civil Society Advisor, USAID Meeting to Review the Interim Report William McKinney, USAID/Azerbaijan; Valerie Ibaan, Humanitarian Response Program Specialist; Gulnara Rahimova, Program Development Specialist; and Evaluation Team FINAL REPORT 84 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Mercy Corps Myuriam Khoury, Program Director; Robert Stocker, EO Program Manager;Kamran Abdullayev, EO Program Officer; Chingiz Mamedov, EO Program Officer; Muhammed Amer Mir, Director of Finance and Compliance; and Mary Hennigan, Health Program Manager FINAL REPORT 85 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ANNEX C REDUCED HUMAN SUFFERING IN CONFLICT-AFFECTED AREAS IN AZERBAIJAN Evaluation Approach and Methodology This evaluation will examine USAID/Azerbaijan’s Strategic Objective 3.1, Reduced Human Suffering in Conflict-Affected Areas, with emphasis on the Azerbaijan Humanitarian Assistance Program (AHAP), which is the major vehicle for providing assistance under this strategic objective. The evaluation will look at AHAP both as a set of activities designed to achieve the strategic objective and as a mechanism for managing those activities. The AHAP is implemented through a cooperative agreement with Mercy Corps International (MCI) which accounts for an estimated 90% of S.O. 3.1 programming. It commenced in January 1998 and, as amended, is a six-year activity with a funding level of $45 million. Currently, all AHAP programming can be categorized as transitional. Activities are implemented through the award of sub-grants to international NGOs (mostly as part of competitive processes) and are concentrated in four geographic regions: Central and South – together comprising the so-called "IDP Belt"; Naxicivan, and the Baku urban area. Activities are funded in three major program areas -- community development, health, and economic opportunity. Program delivery by sub-grantees is scheduled to end in October 2003, which is also the end-date of the current USAID country strategy for Azerbaijan. The evaluation approach follows current USAID guidance as a collaborative activity that is intended to enhance the effectiveness of program transition and implementation by responding to USAID as well as implementing organization management needs. The team will use an iterative methodology to obtain additional data as necessary following the first round of Baku and field interviews and data collection. The evaluation will 1) document achievements and shortcomings against expectations as reflected in the Cooperative Agreement, PMP, workplans and other programmatic documents; 2) review evidence of effectiveness and sustainability and, following analysis of the database, 3) provide findings, conclusions, and recommendations. The team will use a range of methods to collect the necessary information to accomplish the evaluation purposes. The work commenced with a review of key documents that were available electronically to identify issues and structure team assignments. Additional documentation has been provided and will be reviewed. The first week was spent reviewing the electronically available documents and a Team Planning Meeting was held at the offices of Management Systems International (MSI). All three team members arrived in Baku on the evening of September 5, 2002. Mission personnel provided a detailed schedule covering the full four weeks of the work to be accomplished in Azerbaijan and arranged and confirmed the initial orientation with the USAID Mission staff and the partner interviews. The first day, October 7, 2002 was orientation by FINAL REPORT 86 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Mission and Mercy Corps International personnel. The Mission prepared evaluation schedule and field visit itinerary was reviewed, discussed, and revised. Logistics for the field visits and team hiring of drivers and interpreters were covered. The team will develop interview guides according to the Statement of Work evaluation questions for information gathering at various levels. These guides will serve as checklists for the conduct of interviews and for data collection. The team’s interview methodology relies on structured discussions and focused conversations to complement key informant, group interviews, direct observation and other information gathering techniques. The team does not anticipate using any formal survey or quantitative analysis techniques, but will rely on cross comparisons in order to conduct analysis from several perspectives. Interviews will be conducted with MCI and subgrantee personnel in Baku. Field visit interviews will be conducted with on-site subgrantee personnel and selected individuals to assess effectiveness and community/beneficiary participation in subgrantee activities. The team will provide an interim report following the field visits that addresses the Statement of Work tasks. A refined outline of the final evaluation report will be presented and an interim briefing will be provided to Mission personnel. Additional interviews will be conducted and information collected to constitute the final database for the conduct of evaluation analyses. In the final report the evaluation conclusions and recommendations will be presented with reference to the specific findings and analysis that led to them. A thorough briefing will be conducted for Embassy and USAID staff. A draft final report will be presented prior to the departure of the team from Azerbaijan. An initial draft outline of the final report is shown below along with team assignments. Draft Final Report Outline and Team Assignments Title Page Executive Summary Table of Contents List of Acronyms Introduction Purpose (LS) Team composition and evaluation summary (LS) Description of S.O. 3.1 Program and Structure (Team) Evaluation Analysis and Findings (Health, CB; Management, DS; Community Development/Economic Opportunities, LS) Effectiveness Issues/Constraints Sustainability Policy and related issues FINAL REPORT 87 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Conclusions/Lessons Learned (Health, CB; Management, DS; Community Development/Economic Opportunities, LS) Recommendations (Team) Annexes List of Documentation/References List of Interviews/Organizations Evaluation Schedule/Methodology FINAL REPORT 88 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ANNEX D REDUCED HUMAN SUFFERING IN CONFLICT-AFFECTED AREAS IN AZERBAIJAN October 24, 2002 I. Strategic Objective and Evaluated Activities This evaluation examines USAID/Azerbaijan’s Strategic Objective 3.1, Reduced Human Suffering in Conflict-Affected Areas, with emphasis on the Azerbaijan Humanitarian Assistance Program (AHAP), which is the major vehicle for providing assistance under this strategic objective. The evaluation looks at AHAP both as a set of activities designed to achieve the strategic objective and as a mechanism for managing those activities. The AHAP is implemented through a cooperative agreement with Mercy Corps International (MCI). It commenced in January 1998 and, as amended, is a six-year activity with a funding level of $45 million that ends in January, 2004, an approximate 14 months from the completion of this evaluation. II. Background Although USAID commenced activities in Azerbaijan in 1992, USAID/Caucasus/Azerbaijan’s first formal program strategy only started in FY 2000 and was scheduled to end in FY 2003. The program strategy has been extended to January 2004. One of the four Strategic Objectives (S.O.) of this current strategy is S.O. 3.1, Reduced Human Suffering in Conflict-Affected Areas. The USAID budget for Azerbaijan is $28.3 million for FY 2002 ending September 30, 2002. Of the total USAID budget, $10 million is under S.O. 3.1. The three-year strategy is in the process of being updated to reflect programmatic changes made possible by the January 2002 Presidential waiver of Section 907 of the FREEDOM Support Act (FSA) . Although humanitarian assistance was exempted from Section 907 in 1999, AHAP already consisted of predominantly community development and economic opportunity programs at that time and so continued to be constrained by the application of Section 907 in many instances. Under the waiver of Section 907, activities under S.O. 3.1 have been able to work openly with the Government of Azerbaijan (GOAJ) and are enabled to fund projects within the Government. It remains to be determined to what extent USAID chooses to be engaged in direct project support to GOAJ. The Evaluation Team recognizes that the Mission program in Azerbaijan is comprehensive and also includes economic development, democracy and governance, and cross-cutting activities. The current S.O. 3.1 activities could be incorporated into that comprehensive program, according to Mission requirements. FINAL REPORT 89 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation III. Purpose of the Evaluation (Bold indicates material from the Evaluation Statement of Work) The purpose of this evaluation is to review S.O. 3.1, its associated activities, and the performance of AHAP as a management mechanism. The findings and recommendations will inform Mission management and the implementing Partners how effective the current program has been in meeting the strategic objective and where and how possible adjustments could be made to improve program implementation and impact. This Interim Report presents the assessment of the Evaluation Team of the humanitarian assistance programming following approximately twenty-five field visits to health, community development, and economic opportunity projects. These site visits included interviews with community groups organized under the activities of the AHAP Partners and with focus groups with no or little contact with the AHAP programming. Field visits were conducted in the Sumgait area near Baku, the IDP zone of Central and Southern Azerbaijan, and in the Autonomous Region of Naxcivan. All of these areas have been impacted by the Armenian￾Azerbaijani Conflict. On these visits, the Evaluation Team was accompanied by key program personnel of USAID/Azerbaijan. These initial findings, conclusions, and interim recommendations will be refined after briefings with USAID/Azerbaijan personnel. Additional Team analysis relative to improving program implementation and impact after completion of the current AHAP project period that ends in January, 2004 will be conducted and incorporated into the final draft evaluation report to be delivered on October 30, 2002. A briefing also will be held with Mission and Embassy personnel prior to the departure of the Evaluation Team from Baku. The final report will contain detailed discussions on terms that may require clarification within the context of AHAP programming and the evolving Mission strategy. These include, at a minimum, "transition," "sustainability," and "integration." Community development has, at times, for example, been interpreted to mean primarily those organizational activities that resulted in community action groups. “Integration” has political connotations that if not clarified, could be interpreted as integration of IDPs into the local population. The final report will contain a list of definitions. The following discussions on the SOW Evaluation Questions are intended to serve to present the current assessment of the Evaluation Team and to identify areas to be refined for the final report. The final eight day period of the Evaluation Team’s time in Baku will be used to focus on optimizing the degree to which "The findings and recommendations will inform Mission management and the implementing Partners how effective the current program has been in meeting the strategic objective and where and how possible adjustments could be made to improve program implementation and impact." V. Evaluation Questions A. Review of S.O. 3.1: Strategy and Objectives: FINAL REPORT 90 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation 1. Identify gaps (if any) in critical needs which could be filled with available resources. S.O. 3.1 now consists of exclusively transitional activities. Are there critical humanitarian needs not being addressed? In particular, recent surveys are noting widespread stunting among children under five. The GOAJ remains concerned that international donor￾supported feeding programs continue to be reduced. Sub-grantee personnel in the field have indicated that, to their knowledge, the critical humanitarian needs of food and shelter (emergency) are available to all IDPs. There are, however, pockets of greater need, eg. in Naxcivan and in communities with deteriorating infrastructure and a lack of economic opportunities. While medical services, at some level, are available, the quality of and access to health care and pharmaceuticals varies widely. Housing conditions range from unheated railroad boxcars to newly constructed individual family houses with electricity and drinking water. In the conflict-affected areas visited by the Evaluation Team, local non-IDP residents as well as the IDPs in these communities have need for medical services, electricity, drinking water, and improved roads. According to community and IDP leaders that were interviewed, in addition to health and infrastructure needs, school buildings and community centers were high priorities. These were closely followed by agricultural economic opportunity needs (irrigation, salt water drainage, food processing), youth programs (sports fields and gymnasiums), and ceremonial sites for weddings and funerals. 2. Assess which activities of the Economic Opportunity component under S.O. 3.1 should be transferred to S.O. 1.3 of USAID/Azerbaijan’s program strategy. The S.O. 3.1 program contains three broad program areas: community development, health and economic opportunities. As part of the evolving program from relief to development, micro-credit and small business development activities have been initiated under this S.O.. S.O. 1.3, Accelerated Growth of Small and Medium-Sized Enterprises in Targeted Areas, is focusing its activities on regions of greatest economic promise and has thus far not targeted the "IDP belt" where S.O. 3.1 activities are currently being implemented. However, in an effort to emphasize economic development rather than humanitarian assistance, the Mission has made plans to continue to support the most promising of the economic opportunity activities under a new Community Based Development program managed by S.O. 1.3. The evaluation team is asked to assess this assumption and transfer of activity management. The Evaluation Team visited several economic opportunity programs that were implemented under S.O. 3.1 in the "IDP belt" of Central and South Azerbaijan. The development of these projects, primarily for irrigation water and drainage, relied heavily on the community mobilization efforts to organize community members who then focused on community needs and priorities. In most of the sites visited, the communities developed health clinic and/or school projects prior to undertaking economic opportunity projects. This "integrated" approach would seem to indicate that separation of community development and FINAL REPORT 91 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation economic opportunity from humanitarian oriented health, education, youth, and infrastructure (water, electricity, roads) projects may be counter to the expressed community priorities. The program orientations of the Partners affected actual project development. Community needs can differ from region to region, and indeed from community to community. The Evaluation Team’s observation of school buildings with leaking roofs and mud floors and rudimentary clinic buildings with no medical equipment accentuate the need both for respecting community priorities as identified through the community development process, and for assuring that supplies and equipment necessary to the effective functioning of "hardware" investments, e.g., health center and school rehabilitation, are available. The ACDI/VOCA activity, previously under S.O. 3.1, has been reassigned to S.O. 1.3. Separation of selected current sub-grantee organizations that have engaged in community development (IRC) and the implementation of some economic opportunity projects (CHF, SC, and ADRA) could be done, if necessary, leaving those sub-grantees that have been primarily engaged in health activities (IMC, World Vision, and Pathfinder) in S.O. 3.1. What would be lost would be the synergism that has been directed by the USAID Mission. Extensive coordination and cooperation between Partners has been accomplished and substantially demonstrated in the field. This synergism appears, in many of the field projects, to be a necessary precursor to economic development activities. The Evaluation Team observed that there were many community groups trained and encouraged by IRC that worked closely with partner organizations such as CHF that could support the realization of concrete projects of school building and clinic rehabilitation. Additionally, both CHF and ADRA have engaged in community projects that were not primarily economic opportunity oriented. The Evaluation Team recommends that USAID/Azerbaijan consider the development of a structure of strategic objectives that recognizes the accomplishment of an evolving program in the conflict-affected areas from basic humanitarian needs of food and shelter that are being provided by other donors and, to some extent, by the GOAJ. While there is substantial need for economic opportunity programming, the general economy, even in Naxcivan, has been improving over recent years as shown in World Bank economic indicators for Azerbaijan over the past five years. Gross domestic product has increased at high rates and the inflation rate has been substantially decreased. Some economic opportunity projects conducted by other donors, grantees and contractors with specializations such as microfinance could be encouraged and introduced in some community settings in which S.O. 3.1 sub-grantees are operating. The integration of community development and those community priorities of education, health, and infrastructure along with emerging and priority economic opportunity projects is seen by the Evaluation Team as holding out the most effective and efficient approach to achieve the democratic and open market objectives of USAID/Azerbaijan. The Evaluation Team notes that USAID/Azerbaijan has already undertaken the process of defining strategic objectives to develop a strategy statement that reflects the improving humanitarian situation in Azerbaijan and the impact of previous and current USAID FINAL REPORT 92 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation programming. It is recommended that this strategy statement continue to have a humanitarian oriented context. It is recommended that since basic human needs have been met at some level, the revised S.O. reflect the phasing to broad-based community and economic development to targeted communities of greater need and potential as contrasted with S.O. 1.3 target areas of greater opportunity. The required set of RFAs, RFPs and extensions as appropriate should be completed over the next seven to nine months to allow an orderly program development and minimum disruption of correlated USAID current projects by January, 2004. B. Review of S.O. 3.1 Activities: 1. Assess the effectiveness of activities under each of the three major program areas. (community development, health, and economic opportunity). Do the activities as designed and implemented address needs under the S.O. and directly contribute to its attainment? Have they been effective in achieving USAID objectives and intended results? Are the three program areas currently being implemented sufficient and necessary to the S.O.? What areas have been the most effective? What major constraints have been identified? The Evaluation Team observed many project sites where activities in all three major program areas have been successfully attained to the planned objectives of those communities. Accordingly, to the extent that USAID is aligned with community priorities, USAID objectives and intended results have been achieved in these cases. All three program areas, in the sites that demonstrated an integrated and sub-grantee coordinated approach, were viewed as being necessary to addressing the S.O. objectives but in some cases not yet sufficient in their scope, duration or scope. But in those cases where integration was occurring, all three areas were mutually effective and dependent upon each other to achieve the level of effectiveness noted. These successful endeavors also resulted in community organizations that stated that they were sufficiently confident in their own abilities to undertake new projects with and, if necessary, without donor support. The Evaluation Team noted, however, that the additional resources available through the USAID sub-grantees had a high impact on community growth, satisfaction and realization of subsequent projects. The difference of $10,000 in sub-grantee support compared to $20,000 was estimated to be in the magnitude of at least three times rather than double. The one case where the USAID financial support was in the range of $10,000, nonetheless, was greatly appreciated by the community and they were prepared to move onto other projects despite the fact that the school roof still wasn’t replaced and leaked. The constraints varied from project site to project site. In one case, support/approval of a community project was denied by the Executive Com and required appeal to higher levels. In all other cases support including a mix of encouragement, labor, technical assistance, materials, and direct financial contribution was provided by Municipal and/or Ex Com officials. A few mistakes were made in terms of underestimating demand for processed agricultural projects and FINAL REPORT 93 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation steps were being planned to rectify these. The Evaluation Team sees an opportunity for USAID to further engage government officials at all levels. 2. How well are the objectives defined for each sub-activity (sub-grantee)? How successful have the sub-activities been in meeting the needs of the targeted beneficiaries? What activities have been the most and least successful? In the cases observed by the Evaluation Team, objectives were largely well defined for each sub-activity (sub-grantee). According to beneficiaries (who defined the needs and priorities), the sub-activities were more than meeting their needs and expectations. The Evaluation Team is satisfied that the field sites were representative of the efforts of the sub￾grantees. There was only one observed instance where an activity may have had less than well defined objectives and needs may have been met only in marginal terms. Assessing the level or rankings of the sub-activities would require additional data collection. 3. Assess the level of community/beneficiary participation in the sub-activities. is it possible and/or desirable to increase the level of participation? The Evaluation Team was impressed with the community and beneficiary participation which was seen as a result of sub-grantee community development (mobilization) efforts. In these observed instances, the level of participation was seen as high and more than sufficient to achieve the immediate community objectives. This high level of participation was seen as essential for programmatic sustainability. Enhanced Municipal and Ex Com participation, in some cases higher than Evaluation Team expectations, has the potential to result in increased community project planning and implementation. While substantial beneficiary participation was observed at most of the ADRA sites visited in Naxcivan, there was in several instances – most notably, the micro-financing sites – at best limited spill-over into becoming engaged in fulfilling related community priorities in economic opportunity and infrastructure such as water, electricity, and roads. The beneficiaries in Naxcivan clearly had as their primary project-related focus health and microfinance issues. This, the Evaluation Team believes, may less reflect the consequences of the community mobilization training provided to these beneficiaries than the fact that the problems found elsewhere in Azerbaijan are further exacerbated in Naxcivan by other realities, addressed in the next section on “geographic appropriateness.” Participation by IDPs in the community and women was largely appropriate. The Evaluation Team would encourage a higher proportion of IDP involvement in the IDP belt to support increased acceptance in these community settings which may be either temporary, in the currently remote case of resettlement, or relatively long-term. Participation by women was fairly high. In one case, however, the project was largely influenced by male farmers with interests primarily in animal husbandry, agricultural irrigation and agriculture production In only three settings, Camps 4 and 5 in the Southern IDP belt, the World Vision project in Sumgait, and an IDP neighborhood in Barda, was there a 100% proportion of IDP FINAL REPORT 94 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation beneficiaries in the observed AHAP projects. In the rest of the twenty or so sites visited, the proportion of IDP beneficiaries ranged from 10 to 25% in the IDP belt. All project beneficiaries were in conflict-affected areas. No IDPs were observed at the sites visited in Naxcivan and IDPs constitute approximately 1% of the Naxcivan total population. B.3 continued: Assess the geographic appropriateness of S.O. 3.1 activities. S.O. 3.1 activities are concentrated in the "IDP Belt" and Naxcivan. AHAP especially is so concentrated, with only small programs in the Baku-Sumgait area. The "IDP Belt" and Naxcivan were initially selected because they were either isolated from the rest of Azerbaijan (Naxcivan) or areas where extreme poverty combined with the presence of large numbers of IDPs could contribute to a humanitarian or social/political crisis (the "IDP Belt"). Also, recommend on whether to continue work in Naxcivan and that for areas where we aren't working, the team should confer with agencies in Baku to see if there are other areas deemed particularly bad off. Community development needs are nearly universal in Azerbaijan according to the field observations of the Evaluation Team. The geographic appropriateness of Mission activities to follow from the S.O. 3.1 AHAP experience will largely depend upon the extent of resources, both Mission personnel and financial, that can be committed. The Evaluation Team observed ongoing community development needs that would be impacted by S.O. 3.1 type activities in the IDP Belt (Central and South), the Sumgait Area, and in Naxcivan. Mercy Corps estimates of the direct and indirect impact of the AHAP Partnership range from 15% to 50% of the communities in the four AHAP program regions. The revised S.O. 3.1 strategy can target areas with cluster potential in order to maximize program impact on surrounding communities. A balance between economic potential and humanitarian need is recommended. In sum, overall economic conditions, community development, and the nature of USAID￾supported efforts in Naxcivan appear to be approximately two years behind those found by the Evaluation Team elsewhere in Azerbaijan. This appearance is explained, the Evaluation Team believes, by the underlying realities found there. Naxcivan has all of the problems found elsewhere in Azerbaijan. But these have been worsened by two additional constraints. First, Naxcivan is effectively more isolated economically than any other component of the country. With the breaking of the rail link between Naxcivan and "mainland" Azerbaijan, only very high value-added products justify the expense of air shipment or the tortuous and problematic land shipment through Iran and Turkey. The chief products of Naxcivan are agricultural and few have a value added component that meets this threshold. Second, as a result both of the war and its subsequent isolation, the entire population of the autonomous region is war-affected. Naxcivan seems in a number of ways simply to be at a lower level of development and growth than the mainland. The only Mercy Corps-supported NGO grantee, ADRA, did not terminate its feeding program until mid-2001 and the need for direct care emergency services has until very recently driven much of its health activities. FINAL REPORT 95 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Community development has been an ADRA focus only for the last two years; unlike the IDP belt, there are no other Mercy Corps Partners in the autonomous region to help support community development through support for projects other than health and microfinance; and ADRA was unsuccessful in its bid to secure a Mercy Corps grant to support cluster and other project activities which have been key to the community development observed elsewhere. If USAID resources permit funding the kinds of community support projects that have proved so beneficial elsewhere, there is every reason to hope that, on a modest scale appropriate to the size of the autonomous region, significant progress is possible. At the very minimum, the Evaluation Team recommends continuing community development, economic opportunity, and health programming in the existing four regions. Of the four regions, Naxcivan was seen as especially impacted by the lack of access to markets and materials. The Evaluation Team recommends that USAID keep Naxcivan as a high priority area for future programming. The engagement in USAID programming by sub-grantees in addition to ADRA would also encourage the synergism observed by the Evaluation Team in the IDP belt in the Central and South programming regions. Extending S.O. 3.1 programming throughout the country (possibly with demonstration level projects in the North and extreme South) would complete the overall transformation of the AHAP programming from primarily humanitarian assistance. However, given that USAID resources may not be available for such expansion and community needs in the relatively well￾off areas in the North and extreme South are perceived by the Deputy Prime Minister as a secondary priority, the Evaluation Team recommends that emphasis be given to the existing four AHAP regions. A. Review of the AHAP mechanism: Past Performance and Future Prospects 1. Has the AHAP umbrella mechanism been an effective means of designing and implementing activities that directly address and lead to the achievement of USAID’s objectives? Mercy Corps has mounted a coherent and comprehensive range of programmatic interventions directly relevant to USAID’s Strategy under S.O. 3.1 and the achievement of the supporting Intermediate Results. In addition, its operation of the AHAP umbrella meets or exceeds the mandate to Mercy Corps set forth by USAID in the original Cooperative Agreement. Every grant-funded project, regardless of its sectoral focus, apparently has as its center a community organization element that seeks to build sustainable capacity in target communities. This approach responds to IR 3.1.1.2-Communities Organized to Address Self-Defined Needs. Likewise, it is one of the key ways of achieving IR 3.1.1-Vulnerable Communities Better Able to Meet Their Own Needs. FINAL REPORT 96 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation The substantive content of grantee activities, mostly health and economic development, including micro-finance programs, as well as school and road construction, responds to achieving IR 3.1.1.1-Increased Access to Economic Opportunities and Support Services and IR 3.1.1.3 Communities Have Access to Better Quality Services. Likewise, Mercy Corps’ approach under AHAP has effectively moved the umbrella from direct service to developmental programming. The AHAP projects under the umbrella are now providing no direct humanitarian assistance. Most are located in the "IDP belt" and include IDPs as participants with – in most instances – some IDP membership on community committees. 2. Has the management structure of the umbrella mechanism been effective in making program design recommendations, releasing and evaluating RFAs in a timely manner, mobilizing approved activities, and providing effective monitoring of sub-grantees? The AHAP umbrella has been extremely effective in mobilizing appropriate programming. AHAP I RFAs were issued in March 1998, one month after Mercy Corps was awarded the agreement to manage the program. The second round of RFAs were issued for AHAP II in a similarly timely fashion. The selection process itself appears to be fair, transparent and efficient. The process, modeled on USAID’s own procurement procedures, seems rigorous and objective and appears to operate more quickly than most USAID procurements. Respondents are given approximately six weeks from RFA issuance until the deadline for proposal submission. The RFAs themselves are clear and complete, providing specific evaluation criteria and eligibility requirements, appropriate background data, including the appropriate sector strategy prepared by Mercy Corps in its own proposal submission to USAID. 3. How effective has MCI been in implementing a comprehensive and effective monitoring and compliance system? Does the system ensure that high quality data is collected? Does the system ensure that sub-grantees are compliant with USAID rules and regulations? Do MCI and the sub-grantees use performance data to make informed management and programmatic decisions? The monitoring function is the central vehicle for Mercy Corps’ approach to managing the AHAP umbrella. It is combined with a technical assistance function, which it should be noted is not a part of the USAID mandate to Mercy Corps. Mercy Corps has moved toward an integrated approach in AHAP, attempting to bring the full panoply of services in economic opportunity, health and community development to bear on target communities. Reflecting this approach, it has adopted an integrated approach to monitoring, in which a team of Mercy Corps personnel visits grantees for up to five days. In addition to the monitoring visits, each grantee provides reports to Mercy Corps based on some 150 indicators. A number of grantees indicated that Mercy Corps monitoring is burdensome because of its frequency—recently reduced to quarterly from every two months— combined with the depth and duration of visits as well as required reports. They pointed to time consumed not only by the visits themselves but the advance preparation required of them. The Evaluation Team also recommends that the number and type of program indicators be reviewed and reduced. FINAL REPORT 97 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation According to Evaluation Team interviews, Mercy Corps has had wide latitude in determining the particulars of how the USAID’s S.O. 3.1 Strategy is implemented with minimal involvement by USAID beyond approvals of RFAs and grant awards. Further, despite the copious collection of information and assiduous reporting, the evolving program direction emerges more organically from Mercy Corps management and experience than from a conscious effort to analyze and interpret data. This is not to say that there is anything wrong with program direction. It already has been established that Mercy Corps has mounted an excellent program, relevant to USAID’s Strategic Objectives and the needs of target communities. Rather it suggests that program evolution is more dependent on Mercy Corps’ institutional memory and expertise—both in Azerbaijan and at headquarters—than on the use of data. The most striking and most valuable aspect of the AHAP umbrella—and of Mercy Corps’ management of it—is the atmosphere of collaboration that exists among partner organizations. It rarely is found to such a degree and greatly increases the impact and effectiveness of the AHAP Partnership. This level of integration of partner activities is an innovation of the Mercy Corps umbrella. On balance it seems that Mercy Corps’ monitoring is excessive and that the frequency should be reduced to semi-annual. In addition, the technical assistance function should be greatly diminished, since the AHAP Partners are among the most experienced and respected in their fields. The Evaluation Team observed that partner organizations improved their field projects according to real project situations and opportunities. Accordingly, some of Mercy Corps’ resources now committed to monitoring and technical assistance could be redeployed to coordination with USAID which would have greater payoff in terms of programmatic quality. Further, USAID’s minimal involvement in program direction seems a missed opportunity from the standpoint of fine-tuning its implementation of its own strategies. Without damaging the management efficiencies inherent in the umbrella, USAID could profit from more active involvement in determining program direction and evolution. In general, the compliance function is carried out well. Few grantees indicated concerns about lack of information or misinformation nor did USAID indicate major concerns about Mercy Corps’ ability to secure satisfactory compliance. 4. Assess whether there is a value added in using an umbrella mechanism as an implementing vehicle. Analyze the cost effectiveness of any such value added. Assess the cost to USAID of using the umbrella mechanism and compare it to the additional USAID human resource requirements that would be needed in the absence of an umbrella mechanism. To date, the umbrella has been an effective management tool for USAID, which has assigned one individual as the internal management capacity. The umbrella has yielded a high quality and relevant program. It has had the subsidiary benefit of allowing for a broad mix of organizations to operate in Azerbaijan through its effective process of subgrantee RFAs, more so FINAL REPORT 98 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation than may have been possible had USAID been managing all of the S.O. 3.1 activities directly. In addition, and very importantly, the umbrella has fostered a unique level of coordination. Mercy Corps has managed the umbrella very efficiently. Of a total six-year budget of $45 million it is anticipated that more than $37 million will have been awarded in grants, making the annual management cost approximately $1.3 million. Mercy Corps has a low indirect cost rate of 17 percent and is very frugal in its home office management charges. In addition, its salary structures and especially its expatriate allowance structures are significantly lower than those of USAID. Unquestionably, USAID has received good value from Mercy Corps’ management of the AHAP umbrella. Further, the Evaluation Team recommends a narrower mandate for any future umbrella, particularly in limiting the technical assistance function and reducing the monitoring function, to result in fewer expatriates on the umbrella payroll, further diminishing management costs. It is doubtful that USAID’s costs would be lower in managing the umbrella directly. To do so, especially with the numbers of implementers involved, would require at least one additional expatriate staff person in addition to the Humanitarian Assistance Officer now on board and several additional local staff persons. It also would require two or three additional financial management personnel and attendant costs of increased transportation and office space. The real tradeoffs involve programmatic diversity and to some extent quality. Using the $1.3 million it now spends on Mercy Corps management, USAID could mount a creditable program. Direct management would require limiting the number of program implementers in the interest of management efficiency. It also would sacrifice the intensive oversight AHAP has so far enjoyed as well as a large measure of the interagency collaboration—and the resulting synergies—the umbrella has provided. The choice for USAID will be a function of internal priorities. Where do the activities now carried out under S.O. 3.1 fit in the larger scheme of USAID objectives? Do they support other, more important objectives or are they overriding? If the current S.O. 3.1 objectives— especially community development—are not central features of the USAID strategy, the intensive level of oversight provided by the umbrella may be redundant. What are the feasibility and desirability of adding USAID staff? Since additional staff could be deployed in a variety of areas, are they best used on activities now carried out under S.O. 3.1? FINAL REPORT 99 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation ANNEX E AHAP INFORMATION RFAs, Subgrantees, Annual Monitoring Plan, and Mercy Corps Organization Chart LIST OF AHAP SUBGRANTEES IN RESPONSE TO RFAS Adventist Relief and Development Agency American Red Cross Children’s AID Direct CARE CHF International Medical Corps International Rescue Committee Pathfinder Relief International Save the Children UMCOR World Vision LIST OF AHAP RFAs AHAP I (March 1998 to April 2000) Economic Opportunities Sector Health Sector Shelter Sector Integrated Village Project AHAP II (May 2000 to present) Community Development Program Economic Opportunity Program Health Care Program Social Investment Initiatives Integrated Community Development Program FACSIMILE ANNUAL MONTIORING PLAN FOR AHAP PARTNER FINAL REPORT 100 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Objectives of the monitoring Date Comments Review program start-up: 1. Review establishment of project offices 2. Review field project staff July 2000 1. Review establishment of M&E System 2. Review project coordination 3. Review establishment of Economic Recovery Groups & Action Plans 4. Review Core Enterprise (CE)& Secondary Enterprise (SE) Technical Needs Identified September 2000 Review semi-annual report & prepare report to USAID November 2000 1. Review coalition capacity building seminars 2. Review Community Economic Recovery group actions December 2000 1. Review CE activities including · technology enhancement & capacity building · management /marketing capacity building 2. Review SE activities including · technological enhancement & capacity building · management and marketing 3. Review producers/farmer activities · technical capacity building for farmers · management/marketing capacity building 4. Review IDP/Refugee capacity building (WFP) February 2001 1. Review marketing activities including · Intra-regional marketing, · Inter-regional marketing, 2. Review field project staff April 2000 Review Semi-annual report to USAID June 2001 FINAL REPORT 101 USAID/Azerbaijan S.O. 3.1 AHAP Project Evaluation Informati on Officer R. Maherra mov MIS Officer S. HUSEYNOV Health Manager M.Hennigan Eco. Opp. Manager R. Stocker Program Officer (2) R.Mirzayev N. Aliyev Assistant Finance Officer T. Karimova Legal &Technical Officer F. Rajabov Mercy Corps Azerbaijan Azerbaijan Humanitarian Assistance Program Note: Italics font to indicate national positions Program Officer (2) C. Mammadov E. Alimardanov CD Program Officer (2) E. Ismaylov V. Asadova Translator/Interpreter E. Aliyev Finance Manager S. Melikova Admin Assistant A. Abdullayeva Admin Secretary R. Babayeva Drivers (4) H. Afruzi, A. Salayev, S. Sadigov, R Mustafeyev Maintenance Assictant(2) S.Azimova, V. Gasimov Cleaner (2) M. Ibrahimova, S. Mustafayeva SII Program Officer (1) Z. Guliyeva ADCOM CFO Muhammed Amer Mir Program Assistant A. Kerimova Receptionist/Office Secretary K. Salimova CD & SII Manager S. Leonard Program Director Myriam Khoury Chief of Party William R. Holbrook Admin/GR Manager J.SHAKHVERDIIYE