Catholic Relief Services, India United States Agency for International Development (USAID) Assisted Development Assistance Program II Final Evaluation July 2006 DR. SUDESH MUKHOPADHYAY VASUDHA PANGARE DR. B. S. GARG JENNIFER OVERTON Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 1 TABLE OF CONTENTS EXECUTIVE SUMMARY.............................................................................................. H 1 H DAP I H : 1997 - 2001 ............................................................................................................. 7 H DAP II H : 2002 - 2006............................................................................................................. 7 H DAP II FINAL EVALUATION H .................................................................................................. 8 H AH GRICULTURE.................................................................................................................... 9 H History H ........................................................................................................................ 9H Program Description H .................................................................................................. 9H Results as per Performance Indicators H ................................................................... 11H Achievements in the Agriculture Sector H ................................................................. 12H Analysis H .................................................................................................................... 14H HEDUCATION ...................................................................................................................... 19 H HHistory...................................................................................................................... 19H HProgram Description ............................................................................................... 20H HResults as per performance indicators.................................................................... 21H HAchievements in the Education Sector ................................................................... 22H HAnalysis.................................................................................................................... 26H HHEALTH............................................................................................................................ 29 H HHistory...................................................................................................................... 29H HProgram Description ............................................................................................... 30H HResults as per Performance Indicators................................................................... 33H HAchievements in the Health Sector......................................................................... 33H HAnalysis.................................................................................................................... 36H HPROGRAM MANAGEMENT: STRENGTHS AND WEAKNESSES..................... 40 H HSTRATEGY AND PLANNING PROCESSES............................................................................ 40 H HMANAGEMENT STRUCTURE ............................................................................................. 40 H HCAPACITY..................................................................................................................... 42 H HMANAGEMENT INFORMATION SYSTEMS ......................................................... 45 H HMTR FOLLOW UP ....................................................................................................... 46 H HSUSTAINABILITY........................................................................................................ 47 H HPHASE -OUT................................................................................................................... 49 H HCONCLUDING REMARKS......................................................................................... 53 H ANNEXES: EVALUATION SCOPE OF WORK CRS/INDIA ORGANIGRAM INDICATOR TRACKING TABLE Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 2 ACRONYMS AC Agriculture Coordinator AEW Agriculture Extension Worker AFPRO Action for Food Production ANM Auxiliary Nurse Midwife APL Above Poverty Line ASHA Accredited Social Health Activist AWW Aganwadi Worker BCC Behavioral Change Communication BPL Below Poverty Line BS Baseline Survey CBO Community-based Organization CHC Community Health Center CP Coordinating Partner CRS Catholic Relief Services CS Cooperating Sponsor DA Development Assistance DAP Development Assistance Program ECDC Early Childhood Development Center ESF Economic Support Fund FBO Faith Based Organization FE Final Evaluation FFP Food for Peace FFW Food For Work FPG Forest Protection Group FY Fiscal Year GM Genetically Modified GOI Government of India GRPC Girls’ Rights Protection Committee HA Humanitarian Assistance Program HH Household HVY High Yielding Variety ICRISAT International Crop Research Institute for the Semi Arid Tropics ICDS Integrated Child Development Services IDU Injecting Drug User IEC Information Education Communication IFA Iron Folic Acid IMNCI Integrated Management of Newborn & Childhood Illnesses ITDA Integrated Tribal Development Agency IWMI International Water Management Institute MG Mother’s Group MIS Management Information System MOC Missionaries of Charity MSP Minimum Support Price MT Metric Tons MTR Mid-Term Review MYAP Multi- Year Assistance Program NABARD National Agricultural Bank for Rural Development NEC National Education Coordinator Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 3 NFHS National Family Health Survey NGO Non-Governmental Organization NRHM National Rural Health Mission NSS National Sample Survey OBC Other Backward Class OCF Other Child Feeding OP Operating Partner PDS Public Distribution System PHC Primary Health Center PLWHA Person Living with HIV/AIDS PQ Program Quality PSO Partner Support Officer PTA Parent Teachers Association PVO Private Voluntary Organization RBC Residential Bridge Courses RMP Registered Medical Practitioner SC Scheduled Caste SD Standard Deviation SF School Feeding SHG Self Help Group SMCS Safe Motherhood and Child Survival SR State Representative SSA Sarva Shiksha Abhiyan (Education for All Program) ST Scheduled Tribe TBA Traditional Birth Attendant USAID United States Agency for International Development VDC Village Development Committee VEC Village Education Committee VEW Village Extension Worker VHW Village Health Worker WC Watershed Committee WFP World Food Program Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 1 Executive Summary Operating in India since 1946, Catholic Relief Services (CRS) has been in the forefront of combating hunger and providing food support to local partners in the implementation of education, health and agriculture programs. CRS began distributing Title II food in India in 1951 and has been supporting community-based health, agriculture, education and humanitarian assistanceF 1 activities in India for more than 50 years in collaboration with local secular and faith-based grass root organizations. CRS’s experience working with underprivileged groups during the last 50 years shows that the problem of food insecurity can be sustainably addressed through a multi-pronged approach focused on eliminating the root causes of insecurity. Accordingly, CRS has collaborated with USAID, to provide programs in the area of Health, Agriculture, Education and Humanitarian Assistance to food insecure groups. CRS/India’s Title II supported Development Assistance Program (DAP) for 2002-2006 targets 970,000 (annually) of India’s most marginalized communities including Scheduled Caste (SC), Scheduled Tribe (ST), and Other Backward Class (OBC) members, located in the most food insecure tribal belts of central India and the northeastern states. The goal of the program is to improve food security through the participation of India’s most vulnerable groups in sustainable development. In January 2006, CRS/India began planning for an evaluation of this program. Four external consultants were hired to work with the CRS/India team to complete the evaluation: one team leader, who also covered education, one deputy team leader and two sector consultants, one specialist for health and the other for agriculture. The overall objective of the evaluation was to assess and demonstrate the impact of program strategies and interventions and the achievement of the intended results, as measured through indicators developed for each sector. Quantitative data collected during the final evaluation was compared with the baseline estimates and the targets set for each of the indictors. Capacity building is the cornerstone of the DAP II strategy to improve knowledge and practices of household members and increase skills of community- based structures to continue to carry out village based activities on a self-motivated, voluntary basis without the continuing inputs of the program. Institutional development also assists implementing partners so they can become strong development actors in India. DAP II project staff concentrate on training staff at the partner and community level to sustain knowledge. Program implementation ultimately is focused on sustaining the outcomes and results by changing behavior and improving practices in health, education and agriculture. Agriculture The agriculture program uses a watershed management approach to improve the availability of food and food security. CRS and its partners train community-based watershed committees to manage the watershed including the assets that are created by the community to increase water availability used primarily for irrigation. Project technical staff, including agricultural extension agents, work with the watershed committees to carry out soil and water conservation activities and create gravity flow irrigation systems. Community members carry out these activities to create assets, receiving Food for Work (FFW), and contribute to a community fund. Project staff provide learning opportunities so farmers can discover new ways to improve productivity and increase food availability. 1 The humanitarian assistance activities were not included in the scope of work of this final evaluation. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 2 Through the program, farmers have learned about improved farming techniques and applied this newly acquired knowledge, as evidenced by the practices noted in the quantitative data. Farmers have a greater understanding of the importance of high yield variety seeds and the use of manure and fertilizers, as shown by their increased use of these inputs. Farmers have also begun to diversify their crops and plant in both seasons, which is a new practice in the program target areas with little rainfall. Program participants perceive an increase in water availability due to the creation of water surface bodies such as ponds for farming and livestock, as well as other water storage techniques such as check dams. Moreover, there is an increase in soil moisture, which is being sustained through water harvesting structures. Agricultural productivity has increased over the life of the project. Based on the results of the several proxy indicators, there is significant increase in the quantity of land being cultivated and irrigated, while double cropping has also increased almost four-fold. More households are purchasing improved variety seeds and receiving extension support from the government. This increase in extension services demonstrates potential linkages which can be enhanced in view of the pending phase-out. Overall, production was very successful given that program farmers are practicing rain-fed agriculture in dry areas, without irrigation, by using water harvested through watershed management techniques. There is a clear indication that project activities have contributed greatly to certain factors that influence food security at the household level such as increasing on-farm production and income at the household level, as well as providing opportunities for income earning at the village level. In some places, the grain bank concept, in which farmers are storing grain for future use from which they can borrow and then replenish with interest, has been adopted and encouraged. Farmers have a perception of greater food security, and have stated that, as a result of the project, they have more food secure months, with grain available from 5-10 months compared with 2-3 months at the baseline. In areas where agriculture is primarily rain fed, watershed development is now recognized as the most effective way to increase food security and enhance socio-economic development. The characteristic rainfall pattern in these regions makes it necessary to harvest and conserve rainwater for irrigation and drinking water. This two-prong approach should be continued in future agriculture projects, as it is not only the most appropriate strategy but also it is working and producing results. New evidence demonstrates that for sustained impact and success, watershed projects should be implemented for 8-10 years. After only four years, CRS has achieved great improvements in knowledge and practices in the project area, and they should have continued support to engage with these communities so that investments are not lost. Project support should continue for a minimum of three years after the end of DAP II, but five years is preferable. Education CRS is providing educational services to some of the most under-served populations in the country, in extremely remote locations where there are no other educational services being offered by any other private or public agencies. CRS supports 4,894 education institutions and partners and served 306,281 children in Fiscal Year 2005. The primary objective of this support is to increase educational opportunities for disadvantaged children (SC/ST/OBC), especially girls, in quality primary education. As part of the program, Title II food support is distributed to Early Childhood Development Centers (ECDC), School Feeding programs in primary schools (SF), and Boarding institutions (OCF). Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 3 Over the life of the project, CRS has greatly increased access to primary education for children from low-income groups, surpassing the targets set for impact indicators in DAP II. It should be noted that excellent strides have been made to increase school attendance of children living in outreach communities. Moreover, the number of outreach programs have surpassed planned targets. Survival Rates and Net Enrollment Rates have increased dramatically over the baseline and are significantly higher than national rates. Attendance and promotion have both increased over the life of the project; this very positive trend demonstrates that more children are enrolled in school, are attending schools and are staying in school year after year. Drop out rates have reduced in those schools receiving CRS and Title II support; there are few out-of-school youth. Overall improvement can be attributed to many factors: reasonable size of schools leading to availability of adequate number of teachers (school to teacher ratio is 1:5), private management ensuring regularity in teaching and food support for poor children on a regular basis. While initially the school meal may have served to encourage parents to send their child to the school simply for a nutritious hot meal, slowly over the life of the project parents have come to value other aspects of the education program such as quality of schooling and additional support for enhancing achievements. This demonstrates a change in people’s attitude towards education and shows a demand for quality education even among vulnerable households where parents are mostly illiterate and may not have attended the school. This is evidenced not only by the survey results from targeted students but also by the high level of school enrollment among siblings. Mothers seems to be very motivated to send their children to school, and based on the data mothers are seen as the main change agent for children’s education, especially for girls’ education. This implies that mothers should be targeted at the household level to encourage enrollment, regular attendance and linkages with government schools during the phase over period. Linking partners with sustainable government resources such as Mid-Day Meals (MDM) and ICDS for SF and ECDC are important steps to sustain gains made through the investment of Title II resources. CRS should network with like-minded organizations and lobby the Government of India (GoI) to provide food to those students not having access to MDM under Right to Food campaigns. CRS also should advocate for the recognition of some OCF schools by the Integrated Tribal Development Agency (ITDA), Social Welfare Board or Tribal Welfare Board. Health The CRS maternal and child health program began in 1970 when activities aimed at improving the health of women and children were introduced into an existing family feeding program. Currently CRS implements the Safe Motherhood and Child Survival Program (SMCS) using a holistic area-based primary health care approach with community involvement. The program has a strong awareness-raising and training component, carried out by the Village Health Worker (VHW). In addition, CRS established the SMCS center with an Operating Partner (OP) as the anchor of the program in order to promote active community involvement and ownership of the program. The key strategy employed in DAP II is to strengthen access to health services for pregnant and lactating women and their children at the community level such as antenatal care, postnatal care, safe delivery and immunization services by mobilizing communities and referring them to the local government health facilities. In FY ‘05 CRS served 222,096 participants in the Health program and distributed 6,331 MTs of food. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 4 Overall, the SMCS program achieved results by changing knowledge and practices in almost all areas of Safe Motherhood and Child Survival. One of the most consistent improvements seen across all areas of care was the preference of women to seek care/consultation for themselves or for their child at a health facility as opposed to seeking services from non￾medical professionals outside the hospital or health center. More women in the project received Ante Natal Care (ANC) services before delivery and Post-Natal Care (PNC) services after delivery; there was a remarkable increase in the number of women delivering in a health facility. In addition, CRS increased immunization coverage in the target area by 13% amongst all children 12 months of age, and reduced malnutrition by 10% - a noteworthy achievement. There was notable improvement in mother’s knowledge for almost all danger signs during pregnancy, which is important in order to take timely & appropriate action. The knowledge was also significantly improved for the benefits of ANC in general: knowledge considerably improved for the danger signs during delivery & the benefits of delivery by trained personnel. Mothers increased their knowledge about common danger signs during the postnatal period such as excessive bleeding and abdominal pain. Mothers also understood that the benefits of PNC are to identify complications and obtain advice/special care for both the mother and child. The knowledge of mothers on child nutrition improved in the end line evaluation in comparison to base line survey. Mothers were more involved in the process of growth monitoring and they understood how to interpret the growth card accurately; however, compared with the baseline, fewer mothers had growth cards and participated in growth monitoring during the last month preceding the survey, which should be verified and explored. The women also participated in nutrition education sessions especially in the case of growth faltering. Mothers’ knowledge was good in terms of advantages and benefits of immunization and Vitamin A. Although the changes in percentage points in the indicators were modest, the gains made are significant since these are difficult areas in which to change behavior. Moreover, the change noted from the baseline signifies that the strategies used to raise health awareness and increase access to services by forming strong links with the government health services are working. CRS should try to expand the coverage to reach originally proposed target levels in the remaining years under Title II support to consolidate and sustain the change in behavior. Even though CRS will phase out of SMCS under the phase out program, all efforts should be made to expand knowledge and practices to households within the target communities not yet reached by the program in the remaining years. Implementation of program activities supported under DAP II can be further sustained by linking community- based activities with government facilities providing health, education and agricultural extension services. Government programs can also provide technical assistance and other support (please see Sustainability Section). These linkages will provide an opportunity to continue to maintain gains made under DAP II in knowledge and practices and motivate communities to continue volunteer efforts they made under DAP II. Food resources may be sustained only in some sectors, and not in all communities. In some cases, additional cash resources may be available to retain village level staff from either government schemes or private donors. Implementing partners at the OP level are working to establish relationships with national programs and other donors to obtain new sources of funding. The Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 5 vast majority of CPs already has funding for other programs outside of the DAP, some of which is from CRS. CRS has held several strategic planning sessions to consider issues of sustainability in light of the pending phase out of Title II resources from India to further study issues that may not have been considered during the design phase of DAP II. CRS has explored the possibility of building formal linkages with government programs through advocacy; documenting models and best practices and diversifying its funding base to support them; as well as a geographic consolidation and prioritization of program activities to help naturally reduce expenditures as resources from Title II decline. In line with this strategic reflection, CRS has developed a preliminary phase out strategy, which includes many of these themes, and has worked with each of the CRS State Offices to identify preliminary phase out options and strategies during the next five-year period. Although Title II resources have supported CRS food security programs for many years, community based activities began only under DAP I in 1997. Furthermore, DAP II represents a departure from many activities in DAP I and an overall improvement in strategy and implementation in all three sectors. Strategies deployed under DAP II have proven effective as evidenced by the final evaluation. However given more time, CRS could extend these successes to more households, consolidate gains made over the last five years and build solid linkages with government programs to secure the resources necessary from these schemes to ensure continuity of Title II-supported activities. The evaluation team, therefore, is recommending that the program be supported for an additional five years, which would imply a total of ten years of support to these initiatives (DAP II + new program). Brief Recommendations for the Phase Out Program: o CRS should help community structures make decisions about how they will continue program activities when the program ends. o CRS should reinforce community structures and capacities in the next project period. o CRS must ensure that target communities have made formal linkages with local government departments for continued technical assistance and support. o CRS should focus on expanding best practices across all three sectors and discontinue any pilot activities that are unsustainable. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 6 Introduction Despite impressive economic growth, the lives of millions of poor and marginalized continue to stagnate in poverty and malnutrition across India. 300 million Indians still live in absolute poverty: 59% of rural and 48% of urban households do not consume enough calories. Half of India’s children are malnourished: over 200 million children, or about one-quarter of the world’s entire figure. One in eleven children die before the age of five and 2.4 million children die every year from preventable and curable diseases: 46% of India’s children are stunted, a key indicator for chronic malnutrition. As confirmed by USAID itself, the numbers of these hungry children have actually increased over the last 7 years. F 2 It is perceived that India has achieved reasonable self-sufficiency in food grains to meet the needs of its growing population. While at the macro level it appears that there are sufficient food stocks to cover the population’s needs through the Public Distribution System (PDS) designed to reach the poorest sections of society, there are still certain communities and geographic pockets whose needs remain unmet. Food security in these areas is constrained by poor availability, access, and utilization of food. Small size farms, lack of irrigation facilities, unsustainable natural resource management and the recurrence of droughts and floods hamper the ability of marginal farmers and landless laborer groups to produce sufficient supply to meet their family’s annual consumption requirements. The food security context is further affected by declining real incomes at the household level, which limits access by the target population to cereals, high calorie proteins and enriched foods. Food insecurity is exacerbated by poor utilization of food due to limited knowledge of appropriate feeding practices and nutrition and limited access to quality health services. Low literacy levels and gender inequity contribute to the vulnerability of these groups and render those living in the tribal belts of central and northeastern India among the most food insecure populations in the world. Operating in India since 1946, Catholic Relief Services (CRS) has been in the forefront of combating hunger and providing food support to local partners in the implementation of education, health and agriculture programs. CRS began distributing Title II food in India in 1951 and has been supporting community-based health, agriculture and education initiatives in India for more than 50 years in collaboration with local secular grass root and faith-based organizations. CRS’s experience working with these underprivileged groups during the last 50 years shows that the problem of food insecurity can be sustainably addressed through a multi-pronged approach focused on eliminating the root causes of insecurity. Accordingly, CRS has partnered with USAID, to provide programs in the area of Health, Agriculture, Education and Humanitarian Assistance to food insecure groups. In the early 1990s, CRS began to shift its program focus from the southern states to the central and northern states in an effort to reach out to the some of the poorest and most marginalized Tribal and DalitF 3 communities. Through FY 1996 and 1997, CRS developed strategies to transform its support to Maternal and Child Health (MCH) centers and educational institutions from a welfare activity, with resources limited to Title II commodities, to a program to develop human capacity through primary education and community-based maternal and child health care. 2 Figures taken from USAID/India’s Phase Out plan for the India PL. 480 Title II Program and documents from CRS/India. 3 Many Scheduled Castes refer to themselves as Dalits highlighting the fact that these have been downtrodden and abused not only economically but also relates to suppression of culture and more importantly dignity Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 7 DAP I: 1997 - 2001 During the DAP I from 1997-2001, CRS India initiated processes to shift its focus from being primarily a feeding program to a community based human capacity development program. CRS employed a transition strategy in DAP I which moved the CRS supported program from one which supported individual and center-based programs to one which targeted communities and was more village-based. DAP II: 2002 - 2006 CRS consolidated the transition to a village-based approach in DAP II, and CRS has worked to improve coverage and extend its reach more comprehensively in area villages as a result. DAP II changed some of the geographic focus from DAP I, which is explained further in the following sector chapters. Working with a network of at least 67 Coordinating Partners (CPs) and more than 2490 Operating (grassroots) Partners (OPs), CRS focuses its attention on building community capacity to initiate and sustain food security impacts. In DAP II, CRS expanded DAP I operations in small hamlets to the larger revenue village. CRS/India Development Assistance Program for 2002-2006 targets 970,000 (annually) of India’s most marginalized communities including Scheduled Caste (SC), Scheduled Tribe (ST), and Other Backward Class (OBC) members, located in the most food insecure tribal belts of central India and the northeastern states. The goal of the program is to improve food security through the participation of India’s most vulnerable groups in sustainable development. Objectives of DAP II 1. Improve health of 240,000 pregnant/lactating women and children aged 0-3 years o Ensure safe and healthy pregnancies for 90,000 women and improve nutritional status of 150,000 children aged 0-3 2. Increased agricultural productivity of 200 farming communities during the five year DAP period o Increase multiple cropping by improving water security for agriculture 3. Increase opportunities for and participation of 350,075 disadvantaged children (SC/ST/ OBC) annually, especially girls, in quality primary education o Ensure access to primary education for 350,075 disadvantaged children (SC/ST/ OBC) annually, especially girls. o Improve educational quality in 3,535 program schools 4. Provide a safety net to victims of calamities, the destitute, orphaned, sick and dying, especially children o Provide short-term emergency food supplements. o Supplement food intake of orphans, the disabled, destitute and dying poor including HIV/AIDS infected persons and HIV/AIDS affected children, in humanitarian care. An important piece of the overall context for DAP II is the funding situation. For several years, FFP and the local mission warned CRS of impending reductions in Title-II support to India. In early 2004, these discussions became more specific, and CRS, along with CARE (the other DAP Cooperating Sponsor in India) prepared Phase Down plans which were consolidated by USAID and submitted to FFP in June 2004. In this plan, CRS proposed to phase down DAP II commodity levels by 25% by the end of the Phase Down period, by securing resources from local government for programs currently supported with DAP resources. The plan primarily focused on phasing over/phasing out of agriculture and health Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 8 (Safe Motherhood and Child Survival), leaving a core program of education (CRS’s Church partners’ top priority, given the vast network of Catholic schools serving marginalized SC/ST/OBC students with Title II commodities) and humanitarian assistance. This plan was accepted by USAID/India and in April 2005, CRS was instructed by the FFP India point person to move forward with the development of a next phase project proposal based on these funding levels. In early July 2005, CRS/HQ and other Cooperating Sponsors were informed by FFP/Washington that FFP was facing a serious budget shortfall. As a result, CRS/India was instructed to develop a new multi-year development project for FY 2007-2011 which would phase out Title II resources from India completely by the end of the period. Resources for the next project would consist of very limited food support at 43% of FY 2006 totals in the first year, and would progressively reduce thereafter to the end of the project. The USAID Mission has recently submitted a plan for a three-year Phase Out of development activities, with continuing support for humanitarian assistance programs in years four and five. DAP II FINAL EVALUATION CRS carried out baseline surveys in FY’02 to set benchmarks and targets for the Agriculture and Education sectors. For the health sector, the baseline levels were estimated using the DAP I final evaluation results and the USAID funded National Family Health Survey conducted in 1999. The Mid Term Review (MTR) of DAP II was carried out in April 2004. The MTR revealed that CRS was making progress against the implementation plan and had met mid-project targets. However, the MTR also highlighted the linkage between the positive outcomes of the agriculture, health and education programs and household level food security was not sufficiently documented. As per the grant agreement, DAP II was scheduled to conclude by December 31, 2006. However, in April 2005 CRS and USAID mutually agreed to realign the DAP with the fiscal year cycle. Accordingly, DAP II will conclude by September 30, 2006. To ensure objective evaluation and per the approved DAP, an external team was identified to evaluate the CRS/India Title II program from January to May 2006. OBJECTIVES OF THE DAP II FINAL EVALUATION CRS/India initiated the evaluation of its current the Title II program in January 2006. Four external consultants were hired to work with the CRS/India team to complete the evaluation: one team leader, who also covered education, one deputy team leader and two sector consultants, one specialist for health and the other for agriculture. CRS reviewed and finalized the scope of work for the final evaluation and the evaluation team members in close consultation with USAID. The overall objective of this evaluation was to assess and demonstrate the impact of program strategies and interventions on the achievement of the intended results, as measured through indicators developed for each sector (Please see annex for the complete SoW, evaluation methodology, design and sample size). Data collected during the final evaluation was to be compared with the baseline estimates and the targets set for each of the indictors. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 9 Results and Discussion of Findings AGRICULTURE History CRS’s agriculture programF 4 started as a result of the agency’s response to drought in Bihar, one of the least developed states of the country, in 1966. Nineteen (19%) percent of India's total land mass, where 12 percent of the population reside, is drought-prone. However, close to 68 percent of the total cultivated area in the country is drought-prone. During the Bihar drought response, food was used to partially compensate program participants for their unskilled labor in a specific on-farm activity in a number of scattered villages. Due to an identified need and the success of the drought response, the program expanded to other areas of the country. In the initial years, the program maintained the same focus helping individual program participants to improve their resource base by undertaking land development and digging/desilting irrigation wells. A few CRS partners with more advanced technical skills began working with communities to repair old ponds and construct new ponds and check dams for increased community access to water. During this phase, local partners used Title-II resources to support these construction activities, but received little or no technical support or capacity-building from CRS. In 1997, CRS concluded that the impact of this program would be increased if the program implementation was carried out in a contiguous area and therefore re-oriented the targeting and coverage of the program towards a cluster approach. CRS incorporated the water security activities as a pilot intervention under DAP I. Based on the experience in DAP I, CRS decided that a watershed approach could be more effective than the cluster approach and could be a means to withstand the impact of drought and improve results and long-term impact. Under DAP II, CRS adopted a full-fledged watershed approach designed to increase water security within targeted communities in which livelihoods were based on agriculture. CRS learned during DAP I that communities found it difficult to understand the concepts of watershed management and required more time to ensure community-wide participation and build the cohesiveness of the watershed committees. Therefore, CRS added a preparatory phase within the design of DAP II, referred to as the pre-watershed phase, to enhance community mobilization and farmers’ participation and to increase local awareness and participants’ understanding of watershed development. DAP II also focused more on double cropping, increased technical assistance, water harvesting and overall improved on-farm productivity. Program Description DAP II agriculture activities target 45,000 families most in need in areas with a high percentage of rural Scheduled Caste and Scheduled Tribe population, where the majority of families own less than two hectares of cultivable land; less than 10% of cultivable area had access to post rainy season irrigation. Based on these criteria, CRS limited agricultural interventions in DAP II to four focus areas: Jharkhand, centered around Ranchi; Chhattisgarh, centered around Raipur; Bundelkhand, centered around Jhansi in Madhya Pradesh; and Rajasthan/Gujarat, centered around Udaipur. These areas are characterized by low levels of per capita income and human development. Literacy levels are poor especially amongst 4 The use of the word “program” refers to the entire support in agriculture provided by CRS and USAID; the term “project” is used when referring to the individual watershed activity identified by the community at the local level. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 10 women, and infant mortality is relatively high. Local people rely primarily on subsistence rain-fed single crop agriculture and small ruminant-type livestock farming. Throughout most of the year, the residents experience acute shortages of water. High temperatures above 110F are very common during the summer months. Denuded forests, rugged landscape, undulating terrain, with low rocky outcrops, narrow valleys and plains are some of the general topological characteristics of the areas. The rainfall distribution pattern in these areas is very irregular and there are generally drier conditions throughout most of the year, with approximately 90% of total rainfall occurring during the monsoon season (July to September). The watershed activities are community based and initiated upon the request of a village to the Operating Partner (OP), which is approved by CRS. Once approved, the program begins with the pre-watershed phase, during which meetings are held between the OP and the village to form and train the watershed committee (WC). CRS and the CP/OP train the WC to manage all aspects of the watershed including planning, implementation, impact assessment, and conflict resolution. Smaller user groups and self-help groups are formed to manage particular socio-technical components of the program. Women and other socially and economically deprived groups must participate in the WC and users groups, so that their needs are considered as part of the decision-making process. For example, thirty percent of watershed committee members are required to be women. The WC establishes internal guidelines, maps resources institutions and identifies development priorities. CRS and the implementing partners examine the technical feasibility and environmental aspects of the proposed development activities. The WC then prepares an implementation plan that includes soil and water conservation and gravity flow irrigation systems. Community members carry out these activities to create assets, receiving food for work (FFW), and additionally contribute to a community fund. Participants receive 4 kilograms of bulgur and 250 milliliters of fortified vegetable oil per day of work, the value of which is approximately equal to the minimum daily wage. As part of the agriculture program each FFW project is designed to ensure that the participating family contributes approximately 100 days of labor over the ten-month project period of each calendar year. In addition, the community contributes 10% of the allocated workdays for each watershed project. This contribution has greatly contributed to the success of the program, reinforcing the concept of community ownership and enhancing the likelihood for sustainability. The watershed management approach is designed to include an exit strategy from inception, coupled with specific criteria and milestones which permit communities to sustain gains made. Prior to phase out, each watershed committee must meet the following criteria: (1) completed project activities and planned objectives; (2) 40% of the cultivable land double-cropped and irrigated; (3) 70% of community members trained in maintaining and upgrading of assets created during the project and (4) linkages established with other service providers for continued resources and technical assistance after the end of the project. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 11 Results as per Performance Indicators: Indicator Base line Final evaluation Total Total Percent of land double cropped area 8.71 32.37 (target: 31%) Percent of total agriculture land irrigated 5.23 20.12 Percent of Irrigated land double cropped 42.49 65.42 Average area irrigated (ha) per project disaggregated by project duration - 26 percent of target, or 17 hectares (target: 65 hectares) Percent Non -Irrigated land double cropped 6.65 24.05 Percentage of watershed projects with watershed committees having operational guidelines for water distribution and systems to monitor 0 43 (target 100%) Monitoring: CRS was able to meet 75% of its target for food distribution and project participants, serving 200,209 participants, in FY 05 and distributing 19,086 MTsF 5 . One objective of the watershed program is to increase agricultural productivity irrigating more land. Irrigating more land allows farmer to have more cultivable area to plant crops. CRS had planned that each project would have 65 hectares of land irrigated by the end of FY ’05. According to monitoring data collected in FY 05, the average area irrigated (ha) per project was 17 hectares, or 26% of the FY 05 target. However the area under irrigation, in general, has increased from 5.2 % to 20.1 % of total agricultural land. To ensure that results attained under the project are sustained, watershed committees trained by the project should be cohesive, participatory and empowered to manage the watershed after the project ends. These WCs should have well established rules and regulations to guide the use of water and other community assets and systems in place to maintain and monitor the use of such assets. Nevertheless putting such regulations in place is challenging in remote rural locations where large percentages of the population are still influenced by traditional systems and practices. The percentage of watershed projects with committees having operational guidelines for water distribution and systems to monitor these guidelines was 43% in FY ‘05 against an FY ‘05 target of 100%. Impact Production: Increasing the amount of land under cultivation, the amount of land being irrigated and the amount of land being double cropped are all good proxies to determine if production increased and food security improved. The program aimed to increase “overall cultivable area producing more than one crop in a year” from 9% to 31% by the end of the project. Based on data collected in the final evaluation, there was a significant improvement over the baseline in the total amount of land being double cropped in the project area from 5 In November-December 2004, before final FY05 approvals were given and the funding situation was uncertain, CRS/India management advised all State Offices to move forward with a contingency 25% phase down in the agriculture sector. While CRS did ultimately receive full approval of participant levels originally requested for FY05, by that point, significant FFW opportunities during the dry season months of November- February had already passed, thus resulting in a lower utilization rate for agriculture Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 12 8.71% to 32.37% for all lands, both in irrigated (from 42.5% to 65.4%) as well as non￾irrigated areas (from 6.7% to 24%) passing the Life of Activity (LOA) target. Water availability: An increase in the length of time wells retain water is one indication that more water is available to community residents. Increasing water availability in dry areas with little rainfall is tremendously important for both agricultural activities and household water use for both people and livestock. There is a slight increase (from 36% to 39.6%) in the average number of weeks that water is retained in certain wells in the lowland, and a decrease in the average number of weeks that water is retained in the medium upland (25 to 23.1%) and medium lowland (32% to 31%) from the beginning of the project. The total average for both baseline and final evaluation is the same, 31 weeks, which could be attributed to many factors including prolonged drought. Because the project did not support the well construction, the evaluation team did not analyze the wells, their quality, the number of farmers who have wells or their location in the end line survey. Achievements in the Agriculture Sector More than 90% of the participants in the agriculture program were from SC/ST/OBC, 78% reside in kachha homesF 6 and 95% of households do not have any sanitation facilities, confirming that CRS has met it target to reach the most vulnerable populations. Food security can be measured by the availability of food at the household, level of income and rate of migration. There was a slight increase in the availability of food grain in the end line survey from 30% to 36.1% for 91-180 days at the household level. In addition, households reported having grain available for 107 days in the baseline and 192 days at the end line, an increase of 79.4%. Often when agriculture is not successful, members of the family will move elsewhere in search of work. According to data collected in the FE, migration was reduced from 22.7% to 18.8%; additionally the average number of days in a year members of households migrated decreased from 151 days before the project to 84 days at the time of the final evaluation: a 55.6% reduction. This indicates that both frequency and the length of time members of the household migrate have been reduced. Partners stated that the program has helped to reduce migration to a great extent but that increases in yield and vegetation cover were also important achievements. When families borrow money to purchase grain to eat, or borrow grains, it is an indication that the family does not grow enough food to eat, or does not have sufficient resources to purchase food supplies. Over the life of the program, the percentage of households borrowing money from moneylenders for grains declined by more than half from 15.4% to 6.9% and those borrowing grains in general declined from 17% to 10.8%. Incomes from farming increased during the life of the program: more than 70 % of households reported that there was an increase in income from farming. Use of improved variety seeds or practices recommended by the program, such as application of manure, improved crop production techniques and soil quality measures, was also examined. In the FE there was an increase in the percentage of households purchasing high yield variety seeds from the market and from other sources (44.1% compared to 29% in the baseline) and a decrease in the percentage of households buying local seeds. In addition, there is a significant increase in the percentage of households using manure, fertilizers and 6 Kacha homes is an indicator of poverty, without reference to caste; families living in such homes are resource-poor. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 13 pesticides during kharif and rabi seasonsF 7 . The percentage of households not using these inputs has reduced from 12.9 % to 1.1 % in the kharif season and from 80.2 % to 1.1 % in the rabi season. Agricultural Extension Workers (AEWs) interviewed in the FE felt that the increase in productivity was due to the higher use of inputs, such as better seed, and crop diversification. AEWs also responded that soil erosion has been controlled due to land development measures and that as a result the area under cultivation has increased The FE showed a small but important increase in crop diversification. Crop diversification is another proxy to determine increased production and improved food security since it allows households to have varied production throughout the year. Diversifying crops improves food availability in the home during more months and increases income since farmers can better respond to market demands. Based on data collected in the final evaluation, the area being cultivated for cereals, pulses, vegetables and oilseeds has increased from the baseline and new crops have been introduced which were not reported in the baseline survey, including vegetables and fodder. Self-Help Groups (SHG) members and Sarpanches stated that increased productivity was the greatest achievement of the program, followed by crop diversification and better incomes. Sarpanches explained that improved extension services provided by the AEW/VEWs, increased access to inputs such as planting material and access to markets contributed to the increase in crop productivity. Community leaders and farmers interviewed for this evaluation stated that the program has aided communities to bring previously fallow and barren lands under cultivation by managing the watershed area to increase soil moisture and vegetation cover. According to participants in FGDs, cropping intensity has increased and yields have gone up. Linking farmers to government extension services is part of CRS’ exit strategy and one of the key steps to sustainability in this program as it reduces farmers’ dependency on the AEWs and program resources. The number of households who received government extension support increased from 16.2% to 22.9% in the FE, and more farmers (20.8% at FE compared to 9.6% at BS) contacted government staff for agricultural extension services. They sought advice on crop choice (66.7%) and use of fertilizer (53.2%). This data demonstrates that farmers have increased their confidence in government services. There is a significant increase in the percentage of households for whom the grazing location is within the village (37.7% to 72.3%) and a decrease in the percentage of households using adjacent village land (34% to 6.9%), reflecting positively on CRS’ efforts to increase availability and quality of grazing lands. Data demonstrates that there was a decrease in the percentage of children (22.8% to 18.3%) taking cattle for grazing. The watershed program has contributed to water security: 34.7% of households stated that the quality of drinking water has improved after the start of the watershed program, and 54% responded that they changed their source of drinking water. Communities have set rules and regulations, as well as monitoring, to protect surface bodies of water in areas where drinking water is not available. Members of the WC and user groups felt the greatest impact of the program was increased access to and availability of water for irrigation and drinking. They also noted the increase in soil moisture as additional benefit. 7 Kharif means “rainy” and Rabi means “post –rainy” Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 14 Compared to the baseline, more farmers are seeking loans (12.9% at BS to 16% FE), and for those seeking loans for agriculture, more are getting their loans from the bank (14.3% BS to 41.5% FE). This demonstrates that not only is a formal credit scheme available to farmers, but also that farmers are not relying on informal credit such as money lenders, which could be risky. This is confirmed by the fact that there was a decrease in the percentage of households taking loans from the moneylender (38.5% to 25.7%) and cooperative societies (31.7% to 19.7%). More farmers are seeking agriculture loans for fertilizer (46.6% to 51.4%), while fewer people are taking loans for seed purchase (49.7 to 13.7%). Seeds are now more available at a lower cost due to several government programs; CRS is also encouraging seed banks, seed fairs and the storing of seeds from one season to the next reducing the need to even purchase seeds. These efforts combined with greater household income, has increased access and alleviated the need to take loans for seed purchase as compared to the baseline. There is an increase in the percentage of households who are members of groups in the program from 33% at the baseline to 48.6% in the end line: households belong primarily to watershed committees (41.1% to 63.5%). There is also an increase in percentage of groups meeting once a month on a regular basis from 65.2% in the baseline to 77%. There is an increase in the percentage of households where women are members of groups from 28.5% at BS to 45.2% at FE. There has been an increase of women joining SHGs from 68.5% in the baseline to 77% in the end line. Partners also noted that members of groups are meeting more frequently, and evaluation data indicates that 80.6 % of these groups meet once in a month. All watershed committees responded that there has been a social change in the village due to the watershed development program: villagers are taking an interest in village development and there is greater solidarity, mutual support and cohesion among community members, especially those from diverse ethnic groups. Participants in all groups noted that school drop out rate has reduced and more children are going to school. CRS staff are also proud that tribal communities with whom they work have adopted new agriculture practices, and that there is a stronger feeling of solidarity in the communities and a new leadership has emerged. Analysis The qualitative and quantitative data has been provided above; however the following paragraphs will help identify program strategies, structures, systems and interventions that contributed to (or impeded) the achievement of intended impact of program interventions by citing key factors for success and best practices as a result of the program activities. Impact of the pre-watershed phase The pre-watershed phase has contributed greatly to strengthening village-level institutions. During this phase, CRS and partner staff facilitated the development of the WC and gave members time to understand program objectives and how the program functions. Under the Haryali Guidelines, government watershed programs are implemented by PRIs and there are concerns that the Watershed Committees may be less participatory. Sector consultants were impressed with the level of participation in the WCs, however it was noted that the WCs under the CRS program are not recognized by PRIs. Linkages with PRIs will allow WCs to become formally recognized by the local government, which could be an enabling factor in the search for additional resources and participation in government schemes. In those areas where CRS and partners provided close monitoring and greater time commitment, results were outstanding. The program’s investment to build the capacity of the WC resulted in strong performing WCs that are well-organized, able to mobilize community members and complete community assets. WCs hold regular meetings, keep minutes and Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 15 have rules and regulations for how the committee will be self-governed. There is also evidence that these committees have taken important decisions about program activities in the absence of AEWs, OPs or CP staff and have resolved conflicts on their own. The pre￾watershed phase is seen as a best practice and key to success of the watershed management approach, and a positive factor to ensure sustainability. After the completion of the Dadia watershed project in Chhattisgarh, the VDC formed an Education Committee to monitor attendance of students, encourage parents to send their school-age children to school, supervise the mid- day meal program and monitor the quality of education through regular meetings with the parents and teachers. They have also formed a Forest Protection Committee and a Fishery Committee. All these committees were formed by the villagers themselves with very little facilitation by the AEW. The committees meet regularly and keep records of minutes and resolutions. These activities are continuing more than one year after the watershed has exited from DAP II. This demonstrates that CRS’s institutional development strategy is successful and that the gains made by DAP II are able to be sustained. Extension support Agricultural extension support to improve production has been built into the program starting from the pre-watershed phase. Most other watershed programs focus on extension support after the program has been completed, in what is known as the post-watershed phase. However, experience has shown that it is important to build this component into the program right from the beginning. CRS is also making a concerted effort to introduce high-yield seed varieties that are suitable to the region, instead of changing local farming practices to adapt to agricultural inputs that are not suitable to local conditions. For example to facilitate rabi sowing, CRS has introduced the Ashoka variety of paddy, which is a short-duration, high￾yield crop in the kharif season in some of the watersheds. Furthermore, in collaboration with ICRISAT, CRS has introduced rainfed rabi cropping of chick pea in watersheds where it is appropriate to do so, the results of which have been good thus far. In order to make good quality and locally appropriate, fairly priced seeds available to low￾income farmers at the onset of the monsoon season, CRS/India has introduced the concept of “Seed Fairs”. Seed Fairs have been held in at least four states in India and have been widely practiced by CRS in African communities recovering from drought. CRS has created a Seed Fair Manual and has ten years experience implementing seed fairs around the world. The fair motivates farmers and rural entrepreneurs to sell their seeds and other related products and also provides information about agricultural practices and techniques. These fairs have been successful and received a good response from the farmers. Some of the village institutions, OPs and CPs, are also developing seed banks. Improved Knowledge Farmers have learned about improved farming techniques and applied this newly acquired knowledge, as evidenced by the practices noted in the data collection. Farmers have a greater understanding about the importance of high yield variety seeds and the use of manure and fertilizers, as shown by their increased use of these inputs. Increased crop diversification Planting in the rabi season is a relatively new practice in the CRS target areas. It is a difficult practice to adopt since farmers need to plant shorter growing crops in the kharif season in order to accommodate other crops in the rabi season. While this change in behavior and Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 16 practice has started, it is taking place on a pilot basis only and may be too early to evaluate. CRS’s partnership with International Crop Research Institute for the Semi Arid Tropics (ICRISAT) in FY 04-05 led to pilot chickpea trials in five states among 3,400 farmers in 141 communities. Farmers noticed an increase in the average return per hectare where chickpeas, a short-duration crop, were planted as a winter crop in rice lands lying fallow. Furthermore, the fact that farmers are growing vegetables, a water-demanding crop, means that they are generating additional on-farm income since vegetables are a cash crop and grown as a commercial venture. Availability of water A comparison of the data from the baseline and final evaluation survey does not show a significant increase in the availability of water based on CRS’s indicator due to the fact that these areas have been plagued with drought conditions. However, most respondents indicated that a large number of ponds have been created under the watershed projects. Visits to the watershed projects during the evaluation revealed the same; however there is no quantitative data to support this. These ponds have prevented run-off and retained water helping to grow kharif crops even during rainy seasons with uncharacteristically low rainfall; they have also supplemented drinking water for animals and contributed to some household use when other sources were not available. The existence of these ponds is a positive addition to communities without water sources; moreover, Users Groups have created rules and regulations to ensure that the sources are protected from animals, bathing etc. when the sources are used for drinking water. When these ponds are present, households are also able to reduce their expenditures on buying water. Although the data does not demonstrate water availability, soil conditions have changed: soil moisture has increased and is being maintained for a longer period, as evidenced by successful crop productivity. Furthermore, the community itself perceived an increase in water availability. Watershed communities are collecting surface water through water harvesting structures created by the program, such as ponds and tanks. Water is also being harvested in tankasF 8 with silt traps to meet drinking water needs in Rajastan. However it should be noted that according to the evaluation team, number of weeks that water is retained in wells may not be the best indicator to capture water availability; other indicators should be used to provide a fuller picture and demonstrate that more water is available from ponds and dams created by the program. Improved productivity Agricultural productivity has increased over the life of the program based on the results of the several proxy indicators shared above. There is significant increase in the quantity of land being cultivated and irrigated, while double cropping has also increased almost four-fold. More households are purchasing improved variety seeds and receiving extension support from the government. This increase in extension services demonstrates potential linkages that can be enhanced in view of the pending phase-out. Overall, production was very successful given that program farmers are practicing rain-fed agriculture in dry areas, without irrigation, by using water harvested through watershed management techniques. 8 TANKAS: Traditional underground water harvesting and storage tank Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 17 Increased income The program has led to increased income at the household level as evidenced by the data presented earlier in this chapter. The fact that households have increased their income as a result of participating in the program means that households will be more likely to participate in watershed activities even after the program ends. These families may now have funds to send children to school and pay for health care, and to contribute to the village agriculture fund. Contributions to the agriculture fund will help sustain this program by helping to ensure continued resources to maintain community assets, and facilitate technical assistance, inputs and extension services after DAP II ends. Reduced migration Both community members and implementing partners have confirmed that, as revealed by the results of the end line survey, the watershed program has greatly contributed to reducing migration. Because the program provides on-farm income, household members have reduced their need to seek work and earn income away from home. Households now have sources of income in their own village through the agriculture sector reducing the need to migrate. Food security There is a clear indication that program activities have contributed greatly to certain factors that influence food security at the household level such as increasing on-farm production and income at the household level, as well as providing opportunities for income earning at the village level. In some places, a grain bank concept has been adopted and encouraged under DAP II in which farmers are storing grain for future use from which they can borrow and then replenish with interest. Farmers also feel that they have greater perception of food security, and have stated that as a result of the program they have more food secure months with grain available from 5-10 months compared with 2-3 months at the baseline. Although this is a small increase and is difficult to measure quantitatively, it is felt strongly by the communities and is a very positive trend. Nevertheless, families should be food secure all year long, and therefore communities need to be further supported by the program so that these upward trends can continue, be expanded to more households and be sustained. Livestock The availability of fodder has increased: 82.99 % of households produce fodder sufficient for half the year and the largest percentage of households are able to graze cattle within the village itself, due to the program’s efforts to develop grazing lands through improved management of the watershed. Several participants noted that there is a decrease in the number of children grazing cattle; this may mean that more children are going to school, but CRS should further explore this finding. Interestingly, final evaluation data shows that families reduced the number of buffaloes, or work animals, while keeping milk producing animals over the life of the program. This could demonstrate that families sold buffalos due to the prolonged drought, and the need for income. Alternatively, farmers may be selling livestock which are a burden to maintain because they consume high levels of water in order to purchase improved varieties. CRS should examine this point further. Coordination and networking At the state level, CRS is making efforts to coordinate with the local government departments, and network with regional and national NGOs and donor agencies. For example, CRS initiated collaboration with the Citizens’ Forum for Peace Initiative and the Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 18 Resource Agencies Network in Chhattisgarh. In Jharkhand, CRS coordinates with AFPRO for technical support. CRS State Offices are also building relations with NABARD for funding support for watershed projects and Grain Banks. In Jharkhand, CRS has already made progress in their efforts to build ties with NABARD for the SHG program and has begun negotiations to support watershed activities. In Chhattisgarh, NABARD and CRS jointly established a Grain Bank, and NABARD is helping CRS partners to implement watershed projects established under DAP II. NABARD also supports technical training and exposure visits to successful models of SHGs and Grain Banks. CP and OPs are coordinating and networking with government organizations, NGOs and donor agencies. For example In Jharkhand, OP Jamgoria Sevabrata works with several donors to implement natural resource management projects. In FY ‘05 CRS collaborated with International Water Management Institute (IWMI) to fit several watersheds with rain gauges to monitor rainfall. In FY ‘04-05 CRS jointly implemented a pilot project to grow chickpeas with ICRISAT as mentioned above. In Rajasthan, the CP GRAVIS is implementing a project jointly funded by CRS and Wells for India. Technical Suggestions: The following comments provide some additional suggestions to CRS that could enhance the program and capture the successes made by the program. 1. Continue Watershed Development In areas where primarily rainfed agriculture is being followed, watershed development is now recognized as the most effective way to increase food security and enhance socio-economic development. The characteristic rainfall pattern in these regions makes it necessary to harvest and conserve rainwater for irrigation and drinking water. This two-prong approach should be continued in future agriculture programs, as it is not only the most appropriate strategy but it is working and producing results. Nonetheless, watershed management is challenging in drought prone and resource poor conditions. New evidence demonstrates that for sustained impact and success watershed programs should be implemented for 8-10 years. After only four years, CRS has achieved great improvements in knowledge and practices in the program area, and they should have continued support to engage with these communities so that investments are not lost. program support should continue for a minimum of three years after the end of DAP II, but five years is better. 2. Develop specific indicators for rainfed agriculture to improve data collection CRS should develop indicators specific for rainfed agriculture and to measure surface water. The current indicators do not reflect positive gains made by the program in rain-fed agriculture. With simple monitoring techniques and scientific instruments it is possible to measure soil moisture, crops yields and rainfall at the village level. Such information will demonstrate improved productivity and greater food security and capture successes the program had made. New indicators should be designed to measure access, availability and use of various bodies of water created by the program, including ponds to collect run-off water and rain harvesting systems. In addition information about soil quality and other parameters that enhance productivity can be collected and monitored. 3. Establish rules for the usage of water from irrigation sources CRS is working in water scarce, drought prone areas. Through the watershed development programs, water is being harvested for irrigation purposes. The surface water bodies that have been created are also fulfilling drinking water needs. It is crucial to ensure that optimum use Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 19 is made of these resources. Rules have been laid down in all the communities for utilization of sources being used for drinking purposes. Similarly, rules and regulations need to be made for the utilization of water for irrigation purposes, particularly when there is less rainfall. It is also necessary to ensure that water is shared equitably from the ponds and tanks for irrigation or other purposes. 4. Inform farmers of the negative effects of agricultural pesticides Efforts are being made to introduce high yielding varieties of crops. At present rainfed crops are being introduced. But when water is available there is a tendency to grow cash crops which often require more water. Care needs to be taken to introduce crops that are less water intensive and farmers need to be cautioned about this point through the user groups and WCs. The use of fertilizers and pesticides is also increasing. When chemical fertilizers are used more water is also required. Awareness about potential negative effects of the over-use of fertilizers and pesticides also needs to be created. EDUCATION History CRS/India’s support for educational institutions in India dates to the 1950’s when the program consisted mainly of Title-II school feeding for poor and disadvantaged children. Following a sector review in 1995, CRS expanded school feeding to focus on developing human capacity. CRS recognized that while Food for Education Programs could provide a crucial incentive to enroll children in school, the mere presence of the student did not guarantee positive educational outcomes. CRS decided there was a need to introduce a more child-centered, activity-based focus to teaching, and to increase its commitment to girls’ education and other vulnerable groups in communities where inequities existed. Accordingly, CRS introduced a programmatic shift from isolated nutritional inputs to a broader range of complementary educational interventions in 1998, and focused on increasing completion of primary school education (or its equivalent) among traditionally under-served and under-represented groups within primary school populations. The new strategy recognized that engaging community and parental involvement would be essential in order to address constraints on girls’ participation and achieve quality improvements in the school environment. The inclusion of quality and community participation components in DAP I centered on teacher training for Early Childhood Development Center (ECDC) teachers, and community outreach training for community-based primary schools. Innovations for DAP II include the introduction of experimental food-assisted outreach programs to target hard to reach via bridge courses out-of-school children such as child workers and children in crisis situations. CRS and its partners work with parent-teacher associations (PTAs) and village education committees (VECs) to map the educational status of children and engage in community planning to improve regular participation of children in school. DAP II has also continued the “school clustering” concept introduced in DAP I. This approach promotes networking, improved management and sharing of expertise and resources among schools in close proximity to each other. The quality of primary and pre￾primary education provided is a key consideration in DAP activities; considerable resources in DAP II have been committed to training of teachers and administrators. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 20 Program Description CRS supports 4,894 education institutions/programs in some of the most remote areas of the country. The primary objective of this support is to increase educational opportunities for disadvantaged children (SC/ST/OBC) especially girls in quality primary education. There are presently four principal elements in CRS’s education sector program: i) Early Childhood Development Centers, ii) School feeding (SF) in primary schools’ iii) utilizing Other Child Feeding (OCF) in Residential institutions, and iv) Out-reach (OR) in Bridge Courses, described below: Early Childhood Development Centers (ECDC) are village-based pre-primary schools, where children from poor households aged 3-5 years spend a minimum of three hours per day in quality-learning sessions. Teachers of the centers are trained to enhance the learning aptitude among the children using play-based methods. The enrolled children are provided a fortified, cooked, midday meal. In a typical ECDC, the village community takes responsibility for shelter, cooking of midday meals, and contribution towards teachers’ remuneration. School Feeding (SF) programs are located in community-based, private and charity￾supported primary schools (serving children aged 6-14 years) in both rural and urban settings. ST/SC/OBC children are provided cooked mid-day meals. To be eligible for participation in the SF program, schools must offer the state/central government curriculum. CRS provides support for training of selected teachers in these schools, conducted by identified teacher training institutes. Other Child Feeding (OCF) programs are located in selected Residential Institutions and provide supplemental feeding to children are not able to access schooling facilities nearer to their home due to location issues or belong to very poor families or are destitute. The residential institutions are able to cover other operation costs through donations from charitable organizations. CRS provides skills training and capacity-building support to residential institution personnel. Out Reach (OR) programs are located in communities characterized by low enrollment rates and target all school aged children aged 6-14 in a single catchment area to ensure maximum participation in education. DAP II supports several activities in these communities with a goal to enroll out-of-school children in school, keep those children already in school enrolled and follow-up any students with poor school attendance. The project supports the Village Education Committee (VEC) and volunteers to facilitate community mobilization and follow￾up. The program also supports operations and food distribution at residential and non￾residential Bridge Camps in each of the OR communities. The Bridge Camps are accelerated learning centers that prepare out-of-school youth over a 6-18 month period to transit into formal education in regular private or government schools. Profile of Participants The program served 306,281 children in FY‘05. The Final Evaluation (FE) demonstrates that the gender breakdown of participation in ECDC, SF and OCF schools is 56.6% for boys and 43.4% for girls, which has remained roughly the same from the baseline (54.4% and 45.6% respectively), showing that there are more boys at school than girls. The trends for participation according to ethnicity remained essentially unchanged from the baseline with overall participation of ST at 40%, OBC at 31.7% and SC at 20.3%. No significant change has been observed for religion-wise distributions. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 21 Information on participating children’s family members’ is used as a proxy indicator of economic status. About 2.5% of children reported being fatherless and 0.9% being motherless, which is similar to trends from the baseline survey. Interestingly, there were increases in the percentage of fathers who are manual and skilled laborers in the final evaluation data, while fewer mothers are working. Illiteracy among fathers was 33.2% in the baseline and 42.3% in the final evaluation, while mothers’ illiteracy rates increased from 58% in baseline to 67.4%. This increase may be due to the fact that the project re-targeted many of its communities after the baseline in order to reach more marginalized areas. Children reporting having more siblings than in the baseline survey, with slightly more brothers than sisters, and more than 60% of siblings are school aged. School attendance for siblings has increased: 89.5% boys (siblings) of school-going age are in schools and 86.6% girls (siblings) are in schools. This is an improvement over the BS where more brothers (84%) were going to schools as compared to sisters (79%), and demonstrates an overall increase in the appreciation for education at the household and community level. Profiles of Schools and Teachers More than 90% of the schools are co-educational, 2% serve children with disabilities, and 88.3% of schools in the final evaluation were based in rural areas, all of which underscores that the project is serving the most vulnerable children in remote areas. Data collected shows that the ECDCs generally are small facilities; SF and OCF programs are large institutions on average catering to more than 200 children each; such institutions are more likely to be multi￾teacher schools and may not have the pressure of multi-grade teaching. Bigger schools in India tend to have better infrastructure and more resources, as well as less teacher attrition. School accessibility has increased from the baseline survey: 94% of schools are accessible throughout the year and 80% fall within one and a half kilometers of the student’s home, facilitating regular school attendance as reported by the end line survey. Teacher recruitment and attrition was not found to be a significant issue, demonstrating that schools supported by DAP II do not suffer from high teacher turnover, a problem reportedly in government schools that discourages parents and erodes their confidence. Results as per performance indicators Practically all indicators have been attained as can be seen from the table below; Impact Indicator Achievements & Targets Survival Rate Disaggregated by Gender ECDC to Primary 64% overall, 67% for Girls (Target 60%) Primary: class I to class IV 88% overall, 87.5% girls (Target 70%) Net Enrolment Rate CRS outreach program catchment areas by gender 90.3% overall, 91.3% girls (Target 85%) Monitoring Indicators Number of children receiving Title II school meal (306,281) (Target 350,075) Number of education outreach programs implemented 106 (more than the target) Percentage of trained education providers using child centred methods in classrooms 64% (Target 65%; 99% target achieved) Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 22 Monitoring Indicators The education program serves a school meal to each student using Title II food donations: 306,281children received a Title II school meal in FY ‘05 out of an approved level of 350,075 students. In FY ’05 CRS implemented 106 education outreach programs, against a target of 97, each with its own Bridge School. In FY ‘05 64% of trained education providers were using child-centred methods, against a target of 65%. Impact indicators Survival rate, disaggregated by gender: The evaluation team collected data to study the Survival Rate of children enrolled in ECDC and Primary Schools (SF and OCF activities in the DAP) in the year 2002-03 and their movement to Class I in case of ECDC and Class IV in case of SF and OCF children in the year 2005-06. The Survival Rate measures the holding power and internal efficiency of an education system. It illustrates the situation regarding retention of pupils from grade to grade in schools, and conversely the magnitude of dropout by grade. According to the final survey, the Survival Rate from ECDC to Primary School was 64.4% overall, and 66.9% for girls, against a target of 60%, which demonstrates a low drop-out rate and a high level of retention. The data is stronger for the Survival Rate in Primary Schools: from Class I to Class IV was 88.4% for all students and 87.5% for girls: this is a tremendous improvement over the baseline figure of 65%, which is much higher than the national averages. National drop out rates for SC children has been recorded at 41.45% for class I-IV and ST 51.37%. Overall the national drop rates are estimated at 31.47% for class I-IV, and 52.38 for class 1-VII. Net Enrolment Rate (NER) The survey team additionally collected age specific (6-14) enrolment data for the CRS outreach program catchment areas and disaggregated them by gender. Data revealed that 90.3% of children of primary school age in these catchment areas were enrolled in primary school, 91.3% for girls, against a target of 85%. This is a remarkable improvement over the baseline which indicated a NER of only 58% for all primary school aged children in the out reach program communities, and 33% for girls. This demonstrates the program’s ability to enrol out-of-school youth in school through community mobilization and the provision of high quality educational services coupled with a school meal. Achievements in the Education Sector Survival rates across SF and OCF ranged from 85% for ST to 94.2% for other ethnic groups. As stated above the overall survival rate is 88.4%, compared to the baseline figure of 65.4% for class IV, and a national average of 51.4%. The greatest improvement in survival rates was observed for ST children. Boys had slightly better rates than girls. OBC children have better survival rates than SC and ST children but lower than children from general categories. There was a 92% survival rate for those children attending schools found in their village. Accessibility has improved since baseline. Trends are similar for ECDC centers, as reported above. Overall survival rates increased significantly from 51.9% to 64.4%. However, there is a gender disparity with many more girls attending ECDC than boys. It is noted that although more girls are attending ECDCs than boys, families will often send girls to schools which are free, as in the case with the ECDCs; however as noted above survival rates are lower for girls in primary schools where Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 23 parents pay a small fee since traditionally parents are more likely to pay for boys’ education but not girls’ education. Outreach communities CRS programs target select communities – or out-reach locations – in an attempt to blanket the entire population and enroll as many children as possible into school. CRS provides limited support with food for bridge schools and private funds for community mobilization. Once identified, out of schoolchildren have been guided to join available schools. This implies that information given below covers all households in a given outreach community where children are attending any primary school (CRS or non-CRS school). More than 93% of households in the surveyed communities are SC/ST/OBC, with the majority being ST (40.7%) and Hindu (83.8%), and are comprised of mostly (84.7%) nuclear families living in kachcha houses (43.6%). More than 27% have more than 10 acres of land, 37.6% are landless and 60% do not own any livestock. In 62% of the homes, there are two earning family members, however 80.7% of mothers and 65% of fathers are illiterate. Material possession has remained almost unchanged from the baseline. Illiteracy among parents from OR communities is much higher than for SF children’s parents. In the OR areas, 90.3% of mothers reported that children are attending school, against a baseline figure of 58%. It appears that 90% of children are in formal primary schools, 84.3% are government schools, 66.3% are located in their village, and 40.5% are in fifth grade or above; only 8.1 % of children in these communities are attending the CRS bridge courses. This demonstrates that CRS is having an impact on children from the communities who are not attending CRS supported schools. It is encouraging to note that although a majority of parents never attended any school, they are self-motivated to enroll their children in school. When asked about motivation to send their children to school, 73% of mothers replied that they were self-motivated, while 47% replied that they were motivated by their husbands, similar to responses given in the baseline. Although a few mothers stated that they were motivated by the education volunteer or teacher, it is not clear to what extent the volunteer influenced or motivated the mother to send her child to school. As demonstrated in the end line survey results, mothers appear to appreciate education more during the FE than at the time of the baseline, and appear to place more value on the quality of education but it would be interesting to find out what motivated these “self-motivated” mothers to send their children to school. Respondents were further asked about the reasons for choosing a particular school for enrolling their child. Most mothers explained that “close proximity of the school from home” (62.5 %) followed by ‘‘only school in the nearby locality” (30.4%); and “good reputation” (24.5% compared to 11.2% in the baseline) were the reasons. Parent – Teacher interaction is considered important for the continuation of at-risk children in the schooling system. According to the final survey, 39.2% mothers, irrespective of the gender of the child, interact with teachers, compared with 45.4% at the baseline. More than 73% of them interact with teachers at least once a month. For those mothers who did not interact with their child’s teacher, 44% stated they felt it was not necessary; however, this is improved over the baseline of 51%. When asked if their husbands interact with the teachers, mothers replied that 61.1% do, down from 69% in the baseline; when asked why spouses do not interact with the teachers, mothers replied that 60.8% do not think it is important and Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 24 18.7% do not give importance to education. Only 10.1% of mothers discussed their child’s education with the Village Education Volunteer, down from 15.4%. There are still some children in the out-reach communities who are not in school. In the sample survey there were 75 children out of 777 who were not attending school, 35% of whom had been enrolled earlier and dropped out. While there is only a small percentage of children not enrolled in school and the program has made substantial improvement in the NER for these target groups, it is equally important to understand the why these children are no longer in school. When asked why children were not attending school, 50% of mothers replied that the children were required for housework. Of all mothers of children who were not attending school, 34.4% stated they had been contacted and /or asked by a teacher to send the child to school, and 25% had been asked by a volunteer to send the child to school. When asked why those children were never enrolled, mothers replied that the school was too far away (24.1%) or there was no transport (19.6%). Primary School Children from the Outreach Communities In a sample survey of 1,475 children in all primary schools from the OR areas, 40.7 % belonged to OBC, 30.6% to SC and 22.4% to ST communities, and the majority was Hindu (69.2%). Children’s parents worked in agriculture: 31 % of fathers and 34 % of mothers are agriculture laborers, and 73.2% of fathers and 85% of mothers are illiterate. These children have school-going siblings: 53.8 % of children surveyed have school- age brothers and 60.5% have school- age sisters: more than 80% of these siblings also attend schools. More than 92% of the schools are accessible all year around and 81.8% are within the student’s home village. As noted from the information collected at the ECDC/SF/OCF schools, sibling attendance at school is very high, demonstrating a positive spill over effect to enroll more than one child from the home in school. Attendance patterns among these students have increased across all grades. For example, in the penultimate month, attendance increased over the baseline from 72.2% to 74.9% for class I-VII. As expected, school feeding is having direct bearing on the attendance. Children receiving food have better attendance (92%) than their counterparts who are not receiving a school meal (67%). At the baseline, only 20.5 % of children were receiving cooked meals either from the GoI Mid-Day Meals program or from DAP II and in the end line survey, this figure has reached 66.5%. Distance from the student’s home to the school positively influences school attendance: 92.6% of children attended school when the facility was located within one and a half kilometers from their home. Class-wise promotion rates are better than those reported in the baseline survey. The BS reported an overall promotion rate for Class 1-V of 85.2% percent, 86.6% for boys and 83.6% for girls, with an increase at FE to an overall rate of 96%, (95.5% for boys and 96.5% for girls). Nevertheless, it should C hild L abor and Q uality E ducation in A ndhra Pradesh C R S’s State O ffice in A ndhra Pradesh (A P) has been w orking to eradicate child labor in 600 villages spread over 12 districts through a netw ork of 25 church and non church partners since 2002. A P is the 3 rd largest state in India w ith 1.7 m illion children engaged in labor/w ork. C R S efforts aim to abolish child labor and ensure that all children are enrolled in form al education. Program activities include identification of w orking children, increasing com m unity aw areness, and readying children to m atriculate in school. A ccess to schools has been increased through residential and non residential bridge courses and netw orking w ith the governm ent and other existing program s. T hough m ainstream ing is the strategy, tracking these children to ensure continuous participation enrollm ent is also planned. C R S and its im plem enting partners have achieved the follow ing: • E nrolled 17,633 out of school children • R eleased 164 bonded child labors • C reated a partnership w ith M V Foundation to m anage teacher training • B uilt linkages w ith governm ent program s like D PE P, SSA , and N C L P. • Form ed child labor eradication forum of governm ent teachers. • A ddressed related issues of child m arriages. Future directions have now been planned, such as : • L ink partners m id-day m eals and IC D S for SF and E C D C . • L obbying IT D A , Social W elfare B oard or T ribal W elfare B oard to recognize O C F school. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 25 be noted that class promotion is not an accurate indicator for academic achievement, since in most states in India there is a policy of non-detention in primary schools. For those children who attended less than 80% of the school days, an at home survey was conducted to learn why these children were not attending school. As compared to the baseline (41%), 45.2% of mothers indicated that they were contacted for follow-up. More girls were followed up than boys in both cases. In most cases (82%), the schoolteacher contacted the families, compared to 60% in the baseline. The education volunteer contacted only 15.8% of families in the case of non-attendance as compared to 34.4% in the baseline. When asked the reason the child was not attending school, mothers replied: not well (43.5%), household chores (18%) and take livestock for grazing/agricultural activities (8.5%). Overall, 106-education outreach programs were implemented in FY’05, surpassing the target of 97 programs. In FY’05 CRS reported that intensive community mobilization activities brought 27,091 children into bridge courses and formal schools. In addition, many of these children continue and complete classes 7 and 10. In terms of sustainability, the vast majority (80%) of the bridge schools are also linked to National Child Labor Program (NCLP) especially in AP. While food is provided by CRS under DAP II, staff salaries and other administrative costs are covered by NCLP. At times, the government hostels are used to mainstream these children into formal educational institutions, or the community provides support. Such cost-sharing approaches are excellent exit strategies and will enable CRS to phase out support of these schools smoothly. Community members from Mothers’ Groups, PTAs and VEC were interviewed and participated in focus groups discussions about children’s education and the role of different village-level structures. Opinions varied about children’s education: for example, in Andhra Pradesh respondents stated that education is important not only to acquire knowledge but also improvement in agriculture and agriculture related decision-making. The community in West Bengal has a different focus: members here stated that education is an empowering tool for getting equal rights. When it comes to choices between boys and girls, mothers stated “among the son and daughter, I will provide education to the one who is better in studies but I will try to educate both of them”. In Gujarat, it is observed that education is one part of man’s life and without this man is incomplete. Gujarat FG participants also observed education being an equal right for boys and girls and considered 10th standard as a minimum for studies. In general, all community members appreciate the need for education of children (both boys and girls). However, it is clear that economic and social factors affect whether children go to school or not; poverty remains an issue and at times girls and boys are deprived of education opportunities. The program does offer incentives try to overcome some of these limitations, which is further supplemented by the promised to have regular, continuous and relatively good quality education. Activities that reinforce these messages and offer incentives, such as campaigns carried out by VEC, PTAs and Mothers’ Groups, are quite important. For example, in Maharashtra, the VEC has arranged that each child obtains her medical and health certificates, as many of these communities will require certificates to avail the government schemes. These VECs are also responsible for persuading parents to send children, especially girls, to school, for collecting money for teachers’ salary (for ECDC centers) and for organizing the food distribution. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 26 Focus Group Discussion: St. Joseph Upper Primary School Sohkyrmb (Rim) - School Feeding Program The VEC is headed by the Headman and has 7 male and 4 female members. The majority are illiterate but all nodded in affirmative when Headman observed that though they are illiterate, they want their children to be educated. Three female members were interviewed about their families. They each have 5-7 children with 3-5 children in this school. They ensure daily attendance of children in school except for sick days. On the day of the visit there was 100 % attendance at the school. The parents were aware of School Feeding Program, though it is served 20 days a month. School Feeding started about a year ago. Parents volunteer on three- day rotation basis to cook the food for the children. Pre- primary and classes 1-3 run in the morning from 6-9 am: food is served after the school. The other shift is for classes IV-VII from 9.15 a.m. to 4 p.m. In this case food is served at 12 Noon. Parents stated that having the CRS food and support means: • Enrolment and attendance has increased • Children sing and play • Girls also participate in various activities. There is a government school next door, but it does not run properly. Teachers are either irregular or posts are vacant, so parents prefer this school which is managed by the nearby parish. The parish also runs a high school and boarding facility for boys and girls. The mothers’ group is an important community based structure: these groups meet more often than the VEC. For example, in Chhattisgarh mothers reported that they helped the sister in charge with school admission and collecting fees; in return the sister promised that a pond will be made in the village and the children will get good education. The mothers’ group in Andhra Pradesh is quite active: they have been working to increase school enrolment, attendance, academic performance, as well as follow up school dropouts and improve the quality of education in addition to preparing school meals. Analysis The qualitative and quantitative data has been provided above; however the following paragraphs will help identify program strategies, structures, systems and interventions that contributed to (or impeded) the achievement of intended impact of program interventions by citing key factors for success and best practices as a result of the project activities. Program Targeting Based on the data collected in the final evaluation, the education program is successfully meeting the basic goals of DAP II, that is to serve disadvantaged populations from tribal groups and marginalized castes. Socio – demographic data in the baseline indicates that now OR programs are located in areas where migration might have occurred, based on the number of nuclear families, and geographically the program is covering more tribal pockets. This trend is also reflected in the economic profile of families. CRS is providing educational services to some of the most under-served populations in the country, in extremely remote locations where there are no other educational services being offered by any other private or public agencies. This is one of the greatest achievements of DAP II. School Meals Food has played a critical role in the education program since on one hand, it has provided CRS with an entry point into the schools to develop the quality of education being offered by that school, and on the other hand, it has created an initial incentive to families to send their child to the school. While initially, the school meal may have served to encouraged parents to send their child to the school simply for a nutritious hot meal, slowly over the life of the project parents have come to value other aspects of the project, besides the food. Based on Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 27 data gathered in both the household survey and focus group discussions, parents repeated that they value the quality of the education offered by the school. For example, mothers stated that if a government school with a poor teaching reputation and another with better services, such as the CRS supported school, were both located in their village, they would prefer to send their child to the higher quality school. Nevertheless, many parents from low-income families and some implementing partners perceive food as an important factor that allows the project to stabilize the gains made in these marginalized communities. For example, some OPs stated the while the food doesn’t provide an enormous meal to the individual child, it does allow the school to cut its operational costs by up to 50%; and without the food they would need to raise school fees to make up the difference. As long as they receive the food from CRS, they are able to provide subsidies to children from low-income families so that they can waive these fees. Ultimately, food support should continue for at least another five years until the quality of education improves in government schools and families can be convinced to send their children to these schools that now offer the mid-day meals program; or until the GoI provides mid-day meals to private schools serving low-income populations. Behavior change Although schools are more accessible throughout the year and more are located closer to families’ homes, many parents indicted that they are willing to send their children to schools which may be located up to five kilometers away if it is a better school with a stronger reputation. This demonstrates a change in people’s behavior towards education and shows a demand for quality education even among vulnerable households where parents are mostly illiterate. This is evidenced not only by the survey results from targeted students but also by the high level of school enrollment among siblings. As was seen by the evaluators, one positive and very visible quality input was the presence of teachers in all the sites visited. This was also highlighted by the MTR. Another positive point is the quality of basic infrastructure in SF and OCF ensuring class records, adequate school functioning days and hours and learning materials. These factors, coupled with well-trained, dedicated, high￾performing teachers who are not frequently transferred but are more stable compared to government schools demonstrate that the schools supported by DAP II have the quality education parents are increasingly seeking. Mothers’ role in out-reach communities It should be noted that excellent strides have been made to increase school attendance of children living in outreach communities. Moreover, the number of outreach programs have increased, surpassing even the planned targets. Mothers seems to be very motivated to send their children to school, and based on the data mothers are seen as the main change agent for children’s education, especially for girls’ education. This implies that mothers should be targeted at the household level to encourage enrollment, regular attendance and linkages with government schools during the phase over period. Follow-up from schools teachers and volunteers centered around enrolment in the baseline, whereas the concern of teachers was more on regular attendance in the end line survey, demonstrating progress overall. It is also now clear those mothers have an opinion about what education they want for their children, which is quite a departure from the baseline in which parents indicated that they placed less value on education. Parents are more concerned with in- school factors, as seen in the reasons given as to why a small percentage of children still are not enrolled in school. However, the attitude of fathers was slightly discouraging and should be further explored. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 28 Access CRS has greatly increased access to school for children from low-income groups, surpassing the targets set for impact indicators in the project. Survival Rates and Net Enrollment Rates have increased dramatically over the baseline, and are significantly higher than national survival rates. Overall improvement of more than 20 percent points can be attributed to many factors: reasonable size of schools leading to availability of adequate number of teachers (school to teacher ratio is 1:5); private management ensuring regularity in teaching and food support for poor children on regular basis. ECDC survival rates could improve if schools were located within the home village of the child, since, as evidenced by the data, pre-school children will not attend the ECDC if it is outside the community. In the ECDCs food seems to be a key motivating factor to keep very young children in school as well as the follow-up and community mobilization. Attendance and promotion have both increased over the life of the project; this very positive trend demonstrates that more children are enrolled in school, are attending schools and are staying in school year after year. Drop out rates have reduced in those schools with CRS and Title II support; there are few out-of-school children. Quality Education DAP II aimed to address quality by training teachers in child-centered methods. CRS not only tracked the number of teachers trained but also, through classroom observations, observed how many teachers were using these new teaching methods successfully. Results from FY ’05 reveal that 64% of those teachers trained in child-centered methods are applying what they have learned and are using these new teaching techniques as observed and validated during monitoring visits. However, it is not clear why a higher percentage of teachers are not using the child-centered pedagogy if they have attended the training. It was also noted through field visits during the evaluation that not all teachers in the CRS supported schools have completed the teacher training. CRS should explore both of these points further. Technical Suggestions: 1. The education program should move to the next step and begin to focus more on the qualitative aspects of the program While CRS has been extremely successful in increasing access by contributing to a change in the families’ value of education in general, the education program should move to the next step and begin to focus more on the qualitative aspects of the program. Possible steps include improving the quality of teaching and learning, and develop indicators to measure how successful the qualitative aspects of the education activities are. CRS could include designing multiple indicators to measure the quality of teaching as well as indicators on students’ learning, as promotion rates are not a good indicator of grade-level achievement in primary school due to the GoI’s no detention policy at this age. 2. The responsibilities of community level volunteers needs to be further examined in the next phase of the project DAP II supports volunteers under the Outreach Program and Reviewers at the CP Level for monitoring program activities. These staff were active in increasing enrollment and following children out-of-school during DAP II. While these staff will ensure that enrollment and attendance rates continue to remain high in the next phase of the program, the responsibilities of these staff needs to be further examined, in light of the achievements made in access. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 29 3. CRS should link communities to government-supported schools Although the education program under DAP II has created a means for the community to contribute to the activities (cooking food and contributing to the teacher’s remuneration) there is no collaboration with government services built into the program, except for the bridge school in which children, by design of the intervention, are mainstreamed into government schools. Linking partners with sustainable government resources such as Mid￾Day Meals and ICDS for SF and ECDC is an important step to sustain gains made through the investment of Title II resources. CRS should network with like-minded organizations and lobby the GoI to provide food to those students not having access to MDM under Right to Food campaigns. CRS also should advocate for the recognition of some OCF schools by the ITDA, Social Welfare Board or Tribal Welfare Board. 4. Share innovative approaches such as school clustering Innovative activities such as school clustering need to be shared and extended to other parts of the program. School clustering is an innovative, effective, low-cost management approach to enhancing education quality in which teachers are trained in geographic clusters, and trainee groups continue to meet after training, facilitating information-sharing and providing demand-generated supportive supervision on a regular basis. In FY05, CRS partners implemented 50 clusters. 5. CRS should support alternatives for children who cannot afford school fees. The challenge facing the education program is how to sustain the provision of services to marginalized populations. Schools supported by CRS under DAP II will continue to function since they will collect school fees to cover their expenses. However, children from low￾income households will most likely not be able to afford such fees. The following are some options for these children who cannot afford school fees and related costs: o Private schools could offer subsidies or scholarships from NGOs foundations or government schemes o CRS and its partners should begin to prepare families to accept the government schools and visit them together in order to build parents’ confidence in the teachers from the public education system. o CRS and its partners should help the children transition to the closest government school, using some of its experience from the OR programs and bridge schools. o ECDCs could be absorbed by government schools or community programs and then closed: during the final evaluation, the team noted that this was already happening in several cases. o Where government schools do not exist within a reasonable distance, CRS and its partners should lobby the GoI to support the private school with subsidies (e.g. food). HEALTH History The CRS maternal and child health program began in 1970 when activities aimed at improving the health of women and children less than three years were introduced into an existing family feeding program. At that time, CRS partners were distributing food to disadvantaged women and children at centers located in remote areas. Trained nurses who took the initiative to begin introducing health education talks and provision of basic health services during food distribution staffed many of the partner agencies. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 30 In 1995, a participatory strategic planning process led to the formulation of a new health strategy for DAP I (1997-2001) entitled Safe Motherhood and Child Survival (SMCS) program. The strategy involved moving away from clinic-based care for individual needy families, to a more holistic area-based primary health care approach with community involvement. There was a refinement of health education messages and approaches to encourage the adoption of healthy behaviors and improved health outcomes. Aiming to empower women to address their own health needs as well as those of their children and communities, the program incorporated a strong awareness-raising and training component. Central to this was the expanded role of the Village Health Worker (VHW) from a food distribution assistant to community health educator and advocate. Based on the outcomes of DAP I, CRS slightly altered the SMCS approach. While retaining the core community focus, CRS made the SMCS center, administered by CRS’s operating partners at the grassroots level, the anchor of the program in order to promote active community involvement and ownership of the program. Because the DAP I evaluation found that the quality of services provided by the VHWs varied greatly, CRS focused on continuing to build the capacity of the program staff (VHWs) in DAP II. CRS aimed to build and consolidate linkages between the VHW/OP centers and local health service providers for referral purposes in order to strengthen the role of VHWs as community health mobilizers and awareness-raisers, rather than as direct health service providers. CRS also sought to improve coverage within revenue villages where CRS partners already provided some services to a number of hamlets but did not cover the whole village, resulting in an expansion of coverage to all the potential target beneficiary population within the full village catchment area. To increase its technical capacities, CRS collaborated closely with the Linkages Project: Linkages provided training materials for CRS and partner staff on infant feeding practices, including communication strategies to improve mothers’ practices in these areas. Furthermore, CRS phased out its health program in the first half of DAP II, in all locations where there was a potential overlap with WFP and CARE Title-II -supported ICDS programs. Program Description The key strategy employed in DAP II has been strengthening the delivery of health services for pregnant and lactating women and their children at the community level. Some of the highlights of the SMCS program in DAP II include beginning the implementation of a “time￾bound” approach that would limit the duration of CRS support to a participating community and communicating the parameters of an exit strategy with specific criteria for graduation to the community at the beginning of the project. DAP II placed a stronger emphasis on improving access to SMCS services such as antenatal care, postnatal care, safe delivery and immunization services. The strategic objective (SO) of SMCS program is to improve the health of 240,000 pregnant/lactating women and children aged 0-3 years with the sub￾objective to ensure safe and healthy pregnancies for 90,000 women and improve the nutritional status of 150,000 children aged 0-3 years. The SMCS Coordinator at the Coordinating Partner (CP) level is responsible for the overall coordination and implementation of the SMCS program, and the CP implements the activities through Operating Partner: on an average each OP cover 6-8 revenue villages. The OPs have appointed Village Health Workers (VHWs) and TBAs in each of the revenue villages, who are local, literate or semi-literate women. The program provides training to TBAs and VHWs. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 31 VHWs mobilize the communities and carry out growth monitoring and counselingF 9 , but do not provide direct health services. TBAs work with the respective VHW in identifying pregnant and lactating women and children. The VHWs coordinate closely with the Auxiliary Nurse Midwife (ANM) and the ICDS functionary of the village (where there is an ICDS program). Each VHW manages on average 100 participants. There is one supervisor who supports 4-6 VHW. The OP/CP organizes regular meetings with SMCS field staff to discuss activities. The community develops an exit strategy based on criteria set by the program along with the OP and CP staff. The role of VHWs is to carry out social mobilization and increase awareness among mothers and the community about health and nutrition issues related to safe motherhood and child survival. VHWs accomplish this task by providing health education on Health DaysF 10 (also known at the community as MCH day), home visits and counseling sessions. Promotion of health seeking behavior by reducing delay in proper treatment is one of the key messages of the SMCS program. The program encourages mothers to seek consultation at nearby government facilities. The SMCS participants are pregnant women, lactating women and children below age of three years. The program aims to reach economically disadvantaged families in vulnerable SC/ST/OBC communities in targeted areas. The following packages of services are provided to the program participants: • Antenatal care, safe delivery and postnatal care • Growth monitoring coupled with immunization and vitamin A supplementation • Health and nutrition education for mothers, plus home visits • Facilitation of women’s group formation • Village health committee formation • Monthly take-home Title II food rations as an incentive for participation. 9 VHWs provide counseling during growth monitoring sessions on health days as well as home visits for growth faltering children. VHWs maintain daily diary to record such visits and counseling sessions. 10 Health Days are the day in the month during which the program offers growth monitoring and health education. On this same day, the government health agents (e.g. ANM) provide vaccines, ante-natal care and other key MCH services. CRS SMCS Organogram Trained Birth Attendant (TBA) Village Health Worker (VHW) Registered Medical Practitioner (RMP) Village Supervisor (VHW-S) CRS State Office Operating Partners (OP) Coordinating Partner SMCS Coordinator (CP) CRS/ India PQ Health Coordinator CRS State Representative CRS/India Assistant Country Representative Target Community CRS/India Deputy Country Representative Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 32 One community-based activity of the SMCS program is food distribution. Each participant is provided with 1.5 kg of bulgur and 1 liter of oil per monthF 11. This distribution takes place on a fixed day of the month, during which the project offers growth monitoring, antenatal care and children’s immunization services. The majority of participants receive food, but some who do not received food also participateF 12. The coordination among VHW, ANM and AWW has led to a significantly increase in coverage and access to services. ANMs & AWWs participate during fixed day and the government provides vaccines, Vitamin A, IFA tablets. Village health and/or village developmental committees, consisting of 10-15 volunteers, work with the VHW to identify community health problems and solutions related to women and children. A large amount of program data and information is generated at the levels of village, OP, CP and the state CRS office. This information and data collected from the participants by the VHW are recorded in the different registers. The following data registers were maintained on a regular basis i) the pregnant women register, ii) child register, iii) food distribution register, iv) stocks register. Quarterly reports are prepared by OPs and compiled by the respective CPs and transmitted to the CRS State Offices. Data collected at the State Office is entered into a database, using CRS-developed software. Exit criteria The SMCS strategy is designed to include an exit strategy from inception, coupled with specific criteria and milestones that permit communities to sustain gains made. Prior to exit or “graduation”, each community must meet the following criteria: • 80% of pregnant women receive full coverage (2+ injections) of Tetanus Toxoid (TT) • 80% of pregnant women receive 90+ of Iron Folic Acid (IFA) tablets. • Proportion of women with non-institutional deliveries who receive any post-natal service within two months of delivery equal to 50 percent. • 75% of children fully immunized by age one. • 80% of children who receive semi-solid or solid food between 6-9 months of age. • 80% of VHWs, RMPs, TBAs in the village having complete and correct knowledge on the above indicators. • Establishment of linkages with government of India public health centers to provide complementary health services to the community. Profile of Participants According to the FE, 75.7% of participants were SC/ST/OBC. There are on average 6.2 people living in the households interviewed, and 44.4% reside in a Kachcha house; 44.2% of those interviewed use a hand pump for their drinking water, 61.4% store their drinking water in a closed, clean pot, and 72% do not have any latrines. The majority of the mothers participating in the program are aged 20-24 (40.1%) and are housewives (77.5%); 54.5% are illiterate. Only 15.1% were part of a women’s group. Almost 66% of husbands were working in agriculture or unskilled work, and 33.9% were illiterate. There were fewer children 11 Under the approved DAP, the ration size for SMCS beneficiaries was CSB: 2.5 kg, bulgur: 1.5 kg and oil: 1 liter. The bulgur and oil ration sizes have continued to remain the same even after the GoI discontinued CSB. A separate study should be conducted to suggest an appropriate ration combination to complement the recommended daily allowance using only bulgur and oil. 12 The scope of work for this evaluation did not include a comparative analysis of food participants and non￾food participants. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 33 0-11 month-olds (47.2%) in surveyed homes than children 12-23 months old (52.8%); similarly there were fewer females than males. Results as per Performance Indicators Program Participants and Tonnage In FY ‘05 CRS served 222,096 participants in the Health program and distributed 6,331 MTs of food. The coverage in terms of tonnage and food participants was 91% & 93% respectively during year 2005, compared to 79% & 84% in FY 2002, which reflects an increase in coverage over the years. Indicator Baseline Target Final Evaluation 1. % children under 2 years whose weight-for-age falls below - 2SD 47* 36 37 2. Number of VHW, TBA and RMPs trained in health & nutrition 1200 14,400 17,348 3. % children 6-23 m, pregnant women and nursing mothers (up to 6 months lactation) in the program villages, enrolled in the program 20* 83 68.1 4. % program villages that conduct monthly health and nutrition education sessions 44 90 81 5. % registered children under 3 years of age, growth monitored every month 78 90 83 6. % children (12-23m) fully immunized by age one 40* 50 52.8 7. % mothers delivered in the past year who received 90+ IFA tablets 24* 63 36.3 8. % children receiving semi solid or solid food between 6-9 months 70* 75 77.1 *Coverage for SMCS services increased from partial coverage under DAP I to complete coverage of revenue village in DAP II. Therefore, the baseline figures were estimated using DAP I final evaluation, USAID funded NFHS II survey of 1999 and the state wise CRS resources level excepting for the indicator number 8. Baselines for the monitoring indicators numbered 2,4&5 are estimates based on the FY2001 achievements and the number of villages that will phased out in 2001 and number of new villages to be phased in during FY2002 Achievements in the Health Sector It is important that pregnant women make at least three ANC visits during pregnancy and that the first visit takes place within the first trimester. The percentage of women with no ANC visits declined from 34.9% in the baseline to 19.9% in the final evaluation. Better educated women and those who attended health education session in the last month were more likely to have three or more ANC visits. According to the data there was no relationship between age, caste and religion and number of ANC visits. Forty-three (43.4%) percent of mothers registered for an ANC session in the first trimester of pregnancy. There was an increase in the percentage of women who reported seeking ANC services at the DAP-supported OP center (53.3%) compared to the baseline (48.5%). It is recommended that pregnant women consume a minimum of 90 IFA tablets during pregnancy to decrease micronutrient deficiency which could have a negative effect on the unborn fetus. Only 36.3% of women surveyed received 90 or more IFA tablets. A larger Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 34 percentage of women who had attended health education in the last month had received 90 or more IFA tablets. On average women received 78 IFA tablets, an increase from the baseline (69). At the baseline, 78% of women received two TT shots; this figure increased slightly to 81.3% in the final evaluation. While there is a significant improvement in ANC, as noted in the increase in IFA tablets and TT shots, from a qualitative point of view there is need for improvement in the areas of weight measurement, BP measurement and checking the abdomen as well as some other care actions. There is no data collected to confirm if mothers consume IFA tablets or not. At the baseline, 64.6% of women interviewed had received food during pregnancy from CRS, decreasing to 46.8% at the end line. Of those receiving food, the number of respondents who said family members consumed the food declined from 74.7% in the baseline to 64.5% in the final evaluation while the percentage of those who said the food is consumed by the mother and child (the intended consumers) increased from 15.8% to 21.8%. The SMCS project encourages women to seek care at the nearby government health facility when there are complications arising from pregnancy either before or after delivery, and to deliver in a health facility (not at home). While the number of complications during pregnancy stayed roughly the same, the percentage of women who “did not seek consultation” declined from 31.7% to 20.9%, and more mothers did seek consultation at the government hospital increasing from 13.7% to 23.2% in the end line. There is a dramatic increase in the number of women who delivered in a hospital (from 19.6% at BS to 35.9% at FE), and a decrease in the number of home deliveries. Mothers with a higher level of education and those with more contact with health professionals during ANC visits were more likely to deliver in a hospital. Women with complications during delivery increasingly sought treatment (48.7% BS vs. 67.7% FE), and prefer to consult health professionals, in district health facilities. Post-natal care (PNC) is very important for both mother and baby, and the three visits should take place within six weeks after delivery. There is a considerable increase in mothers receiving post-natal care, from 25.1% at the baseline to 35.9% at the end line, with more postnatal checks ups taking place at the district hospital/CHC/PHC and during the Village Health Day with the ANM. All services offered during the PNC visit also increased in the end line. On average, 35.9% of mothers had their PNC visit within the first six weeks after delivery, a significant increase over the baseline figure of 25%. There was a slight increase in women experiencing post natal complications. However, 62% of these women sought consultation, compared with only 50.5% at the baseline. An increased percentage of women preferred consulting someone at the hospital, 41.5% of whom were referred by project staff. Breastfeeding should begin within one hour of delivery and nothing other than breast milk should be given to the baby until 4 months of age. Early initiation of breastfeeding (within one hour) after delivery increased from 10.8% at the baseline to 19.6% at the end line survey. Additionally, 52.3% of mothers gave nothing to their child before initiating breastfeeding compared to 37.6% at the baseline. Colostrum is the first milk which comes from the mother’s breast after delivery and is high in antibodies; nevertheless in many cultures it is taboo to give this special milk to the newborn. The number of mothers expressing (throwing away) colostrum declined from 54.5% to 45.1%. The percentage of babies “not weighed” at birth decreased from 51.6% in the baseline to 25% in the end line. A greater percentage of mothers reportedly sought treatment for sick children (78% to 85%) at the district hospital or Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 35 from a private doctor in the end line survey. The proportion of low birth weight babies was 14.6% (compared to 16.3% at the baseline) against the National average of 30%F 13. Children should be introduced to semi-solid foods ideally at the age of six months after a period of exclusive breastfeeding; however in this program mothers are counseled to exclusively breastfeed for three months and then introduce semi-solid foods between the ages of six and nine months. Seventy-seven (77%) percent of mothers of children aged 6-9 months stated that their children were already receiving semi-solid foods compared to 70% at the baseline. For all the children currently receiving semi-solid foods, 10.7% of mothers introduced these foods after 11 months of age. The number of children surveyed in the final evaluation “ever weighed” declined from 82.4% in the baseline to 79.7% in the end line survey. Overall the number of children with <2 SD malnutrition decreased from 47% to 37%, a dramatic decrease of 10% in malnutrition which is quite difficult to achieve in similar nutrition programs in other countries with similar socio-economic patterns and health seeking behaviors. All children should receive a full complement of vaccines against common childhood illnesses by the age of one year. Vaccine and Vitamin A coverage in the CRS SMCS program rose during the project period, surpassing the LOA targets. Immunization services were arranged at the village level by OPs in collaboration with the ANM from the PHC on a monthly basis at the SMCS center. The vaccines were brought by the ANM, through a collaborative relationship between the local health system and the OP. The proportion of mothers who had a vaccination card was 53.2%. Compared to the base line data, there was remarkable improvement in children receiving measles vaccine, all vaccines & Vitamin A solution. For example, the percentage of children vaccinated by 12 months for all vaccines rose from 46.4% to 52.8%; measles coverage rose from 56.7% to 61.7% and Vitamin A coverage rose from 50.1% to 67.7%. Mothers’ knowledge about the danger signs of pregnancy and benefits of ANC, including IFA tablets & TT immunization in particular, and the danger signs during delivery and benefits of safe delivery increased significantly. For example, in the baseline survey, 32.3% of mothers knew that swelling of hands/feet was a danger sign, compared with 42.4% at the final evaluation. Overall, mothers increased their knowledge about possible danger signs in the postnatal period, and the benefits of PNC. Knowledge about signs and symptoms for diarrhoea has improved: more mothers understood that passing watery stools was a sign of diarrhoea (58% at BS to 70.2% at FE). More mothers knew that ORT was for re-hydration: 20% in the baseline compared to 38.5% in the FE survey. Fewer women had a growth card for their child when compared to the baseline (63.8% vs. 37.6%). Yet those with a card had increased their knowledge about how to interpret the growth card data: 39.2% could correctly assess the nutrition status (normal vs. below normal) compared to 34.2% at the baseline. This excellent understanding of growth monitoring may be linked to mothers’ concern for proper nutrition and could help explain why malnutrition rates declined. The percentage of mothers who knew that Vitamin A prevents Night blindness rose from 21.8% to 34.5% 13 Ministry of Health, UNICEF. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 36 Analysis The qualitative and quantitative data has been provided above; however the following paragraphs will help identify program strategies, structures, systems and interventions that contributed to (or impeded) the achievement of intended impact of program interventions by citing key factors for success and best practices as a result of the project activities. It was observed that the households were divided equally into Nuclear and Joint/Extended family which reflects that in rural area also there is a shift to the nuclear family system. While the average household size was 6.2; the majority (52.7%) had 6 or more. The fact that 80% of the survey participants were SC, ST and OBC shows that the target population meets CRS’s criteria. Limited access to safe drinking water, low coverage of toilets and use of hand pumps, may not be safe and could lead to an increase in water-born diseases. Health problems like respiratory illnesses & vector borne illnesses are high in such situations and pose economic burdens on the household as the out of pocket expenditure on health increases along, bringing down socio-economic status. The SMCS program does not currently focus on hygiene and sanitation. The background characteristics of women reveal that only 15.1% of women were members of saving groups. As the SHGs are highly successful and should be further encouraged to empower women, increase household income and develop sustainable programs. Safe Motherhood Overall, the SMCS program achieved results by changing knowledge and practices in almost all areas of Safe Motherhood and Child Survival. One of the most consistent improvements seen across all areas of care was the preference of women to seek care/consultation at a health facility as opposed to seeking services from non-medical professionals outside the hospital or health center. Although the gains made were small, in terms of percentage points, they are noteworthy since these are difficulty areas in which to change behavior. Moreover, the change noted from the baseline signifies that the strategies used to raise health awareness and increase access to services by forming strong links with the government health services are working. CRS should try to expand the coverage to reach originally proposed target levels in the remaining years under Title II support to consolidate and sustain the change in behavior. Even though CRS will phase out of SMCS under the phase out program, all efforts should be made to expand knowledge and practices to households within the target communities not yet reached by the program in the remaining years. It appears that in spite of the SMCS program there is still a percentage (19.9%) of mothers who did not receive ANC, although this figure declined dramatically from the baseline figure of 34.9%. While mothers with higher education sought ANC earlier, the program should encourage all women to seek ANC in the first trimester and to have at least three ANC visits prior to delivery. Poor coverage in ST was noted and should be further explored. The proportion of ANC services received both at home & center together has increased, especially at the center where the number rose to 51.4% which is a good reflection of the program’s efforts to guide mothers to the center for ANC. There is increase in health providers visiting the home for ANC services, but more women are now visiting the CHC/PHC & village clinic to receive ANC. Many mothers reported receiving ANC by the ANM, demonstrating one Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 37 of the many positive impacts of the MCH day at the village level and collaboration with the local government health system which should be further encouraged. The percentage of mothers who received adequate IFA tablets in the end line (36.3%) did not meet project’s target and a strategy to increase this coverage should be developed. Nevertheless, mothers who received health education recently were more likely to have adequate IFA coverage and TT immunization. The supplementary food during pregnancy provided pregnant women an opportunity to have TT but it is not clear why they did not receive IFA tablets. The evaluation team feels that the poor IFA coverage was due in part to a break in the supply chain of IFA tablets as provided by the government health facilities. While more pregnant women are receiving the required number of TT shots and a higher number of women are going for ANC, some aspects of the quality of ANC as a whole could be improved so that more health practitioners are taking action to complete the weight measurement, BP measurement & abdominal examination. CRS should initiate a quality assurance program in the village clinic and for ANC provided by OP staff. Families who received Title II food under CRS’s SMCS program were also asked if they received food from the ICDS program. The proportion of mothers/child who were currently receiving food from DAP II at the time of the survey was 56.8%; the proportion of families receiving food from ICDS was 38.4%. Further analysis is needed to determine what number of SMCS beneficiaries are receiving food from both SMCS & ICDSF 14. Beneficiary contributions increased during the project. Participants cited that their contributions were towards the cost of food & its transport. Very few responded that the contribution was for VHW salary or for the cost of medicine. The project should make community members aware that these funds are to pay for the VHW honorarium in order to develop a means to sustain SMCS activities after Title II. The evaluation findings show an increase in the proportion of institutional deliveries in comparison to the baseline, which reflects positively on the efforts of the SMCS program. It appears that only female education has some effect on this trend. In the future, the project should further promote institutional deliveries & should mobilize communities for home deliveries only by trained personnel. There is significant improvement in terms of PNC as 35.4% mothers received PNC at end line in comparison to 25.1% at base line. Additionally the data shows that mothers have accepted the ANM as a PNC provider along with the doctor. As part of the SMCS program, one of the key messages promoted by the program was referral to government health providers. Throughout the program, VHWs and other program staff continued to promote the use of government health providers for health care services in health education sessions and other points of contacts with mothers. The fact that the data in the end line survey demonstrates that more mothers are using government health providers for health services clearly signifies to the evaluation team that CRS has done a commendable job of establishing the linkage between the community and the government health system. 14 It is not possible to extract data from the end line survey to show which participants were receiving ICDS rations. However, the evaluation team has recommended that an additional survey could be deployed to obtain such information. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 38 Child Survival The practice of breast feeding is quite high, with the practice of initiating breastfeeding within the first hour after delivery increasing and the practice of giving pre-lacteal feed declining. Similarly, fewer women expressed their colostrums compared to base line, which is very positive. However the practice of introducing complementary feeding proves to be an area of concern, and more progress should be made in this area so that complementary food are introduced in a timely manner – neither too early nor too late. Although there was little change in the number of neonatal complications, with main complications due to low birth weight, difficulty in breathing and convulsions, it is clear that mothers do not recognize the OP staff as care providers as only 2.3% of newborn services were provided by OP staff. The project must review its strategy in terms of new born care as neonatal mortality represents the main proportion of the IMR. In home deliveries, mothers should be advised to seek care from the OP staff immediately in the case of neonatal complications because in many remote locations health facilities are not available. The main causes of morbidity in the two weeks prior to the survey were fever, cough/cold & diarrhea. The majority of mothers preferred to seek treatment from private hospital/private doctor. Although the majority of mothers continued breast feeding in cases of diarrhea a good proportion gave less than usual feed & fluid. In this case VHWs should enhance health awareness and counsel mothers on this point. The practice of monthly growth monitoring decreased and fewer mothers had growth monitoring cards for their children in end line survey. This is again probably is due to the fact the not all children of the villages are enrolled in the SMCS program which has resulted in lower percentage weighed amongst the general population at the time of end line survey. The final evaluation data from FY ‘05 demonstrates that only 68.1% of children and pregnant mothers living in the village are enrolled in the program (please refer to the table on page 36). There was remarkable improvement in mother’s knowledge for almost all danger signs during pregnancy which is important in order to take timely & appropriate action. The knowledge also significantly improved for the benefits of ANC in general and of IFA tablets & TT immunization in particular; knowledge considerably improved for the danger signs during delivery & benefits of delivery by trained personnel. Mothers increased their knowledge about common danger signs during the postnatal period such as excessive bleeding and abdominal pain. Mothers also understood that the benefits of PNC are to identify complications and obtain advice/special care for both the mother and child. The knowledge of mothers improved in end line evaluation in comparison to base line survey for child nutrition: mothers were more involved in the process of growth monitoring however still fewer mothers had growth cards and participated in growth monitoring during last month preceding the survey, which should be verified and explored. The women also participated in nutrition education sessions especially in the case of growth faltering. Mothers’ knowledge was good in terms of advantages & benefits of immunization & Vitamin A. While mothers increased their knowledge on the purpose of Vitamin A (21.8% to 29.7%) still 60% of mothers do not know the correct reason to take Vitamin A. While good progress has been made, more work needs to be done on health education in this area. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 39 Technical Suggestion: 1. Newborn Care should be strengthened utilizing the IMNCI Guidelines. The following measures may be taken: During home deliveries, TBAs are very busy and could be assisted by the VHW. In this case the role of the VHW could be modified so that she is trained in Newborn Care using a skills￾based training and the IMNCI manual. In this modified role, the VHW could also be provided basic tools such as an infant weight machine (Salter Spring Type), thermometer, disposable Mucous Extractor and a warm bag along with the tablet form of paracetamol. The VHW should also be provided guidelines for appropriate referral and IEC materials. At the same time, CRS should undertake formative research to document the harmful and good practices prevalent in community for newborn care in order to develop appropriate BCC strategy. 2. Child nutrition and growth monitoring should be improved. CRS should address anemia in children by providing all children aged 1 - 3 years IFA and deworming at 6- monthly intervals, along with IEC materials to include iron rich food in the diet of child. DAP II recommends that every child between 6 months and 3 years receive 5 Doses of vitamin A solution, and an IEC program on vitamin A rich food should target caregivers. However in some cases only three doses are given to children due to in some cases limited knowledge by staff and poor supply chain management by the GoI. Program standards should be reinforced through refresher trainings, updates and meetings and increased monitoring by implementing partners. In view of new findings in NFHS - II and other evidence, Despite significant achievement in reducing malnutrition by 10%, the prevalence of malnutrition is still quite high and could be reduced by using the TIPS method (Trials of Improved Practices). The project should also reinforce the VHW’s skills in nutrition counseling, and ensure that OPs have Salter scales and sufficient growth cards. Families should be motivated not to give ‘Ghutti’ and Gripe water before six months which is a common practice especially in the tribal areas. Similarly, complementary feeding may be further promoted by developing the appropriate IEC tools for mothers to ensure that the child receives the complementary foods rich in calories at 6 months and not later. 3.CRS should continue to promote institutional delivery and enhance linkages between the PHC/Sub Centre and Trained Dai’s. To plan for and ensure access to Emergency Obstetric Care (EOC), CRS will continue to promote that families arrange transport for planned institutional deliveries. Because 60% of women do not deliver in institutions, the VHC or the PRI should identify a nearby facility and made an arrangement whereby the facility will provide emergency transportation and EOC, for those families that have not made other arrangements independently for identified high-risk pregnancies. The VHC capability should also be enhanced by providing adequate training on postnatal care, so that pregnancy complications may be prevented/managed on a timely basis, in order to bring down the maternal & neonatal morbidity and mortality rates. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 40 External Factors: There are several external factors beyond CRS’s control that have negatively influenced project implementation, and may have interfered with achievement of certain targets. Transportation and supply chain issues have affected program coverage in certain places as noted in the Results Reports. This has hampered CRS’s ability to reach targets for food distribution, number of participants as well as growth monitoring which occurs during food distribution activities. In addition, flooding and other emergencies disrupted supply issues which impacted food distribution targets. Program Management: Strengths and Weaknesses STRATEGY AND PLANNING PROCESSES The DAP II approach The learning from DAP I was incorporated into the planning and implementation of DAP II, resulting in a shift in strategic direction from a welfare approach to a holistic developmental approach. In DAP I interventions were piecemeal, either for creating community assets or in response to the needs of individuals, and in contrast, DAP II focuses on the community. This approach is more appropriate for long-term sustainability so that the project has afforded the opportunity to build and strengthen institutions at the community level. CRS’s strategy also focuses on community ownership, which is ensured by involving all members of the village in the planning and decision-making process and witnessed by the contribution of voluntary labor and cash. Communities have created village level institutions that are empowered to make decisions and take a leadership role about the management of project activities. Appropriate decision making processes at the village level have been introduced in all three sectors, the details of which can be found in the sector reports. MANAGEMENT STRUCTURE Until October 2003, CRS/India co-ordinated its programs through a country office in Delhi, four Zonal offices at Lucknow, Calcutta, Mumbai and Hyderabad, and select State Offices at Bhubaneshwar, Bhopal, Guwahati, Jaipur, Raipur and Ranchi. In order to improve and strengthen partnership and to facilitate decision making, this three tier structure was replaced by a two-tier structure. A national office in Delhi (along with staff working at national level located in the Hyderabad office) coordinates CRS/India’s activities through 12F 15 State Offices. The State Offices work directly with 67 larger NGOs called Coordinating Partners (CPs) in 23F 16 states (including the seven north-eastern states) and the union territories of Andaman and Nicobar Islands, Chandigarh, Dadra and Nagar Haveli and Pondicherry. CPs in turn co-ordinate with more than 2,500 grass root level NGOs or OPs. CRS/India’s DAP II is implemented in 20 states including the seven North Eastern states and the union territories of Chandigarh and Dadra and Nagar Haveli. However, the number of states covered for individual sectors vary. As with all CRS/India programs, a team comprised of both international and national staff manages the CRS/India DAP. A Country Representative (CR) leads this team with the assistance of two Deputy Country 15 However, DAP II programs are implemented in the states covered by 11 State Offices. The Chennai State Office does not oversee any DAP II activities. 16 Andhra Pradesh, Assam, Arunachal Pradesh, Bihar, Chhattisgarh, Gujarat, Himachal Pradesh, Jharkhand, Karnataka, Kerala, Madhya Pradesh, Maharashtra, Manipur, Meghalay, Mizoram, Nagaland, Orissa, Rajasthan, Tmil Nadu, Tripura, Uttar Pradesh, Uttaranchal and West Bengal. Please note that DAP activities do not cover Tamil Nadu, Karnataka, Kerala, Pondicherry and Andaman and Nicobar Islands Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 41 Representatives (DCRs) based at Delhi and Hyderabad, an Assistant Country Representative for Management Quality (MQ) and the department director of Internal Audit. Two Assistant Country Representatives (ACRs) for Program Quality (PQ) assist the Deputy Country Representatives and 12 State Representatives (SRs). The ACRs-PQ oversees a team of National Coordinators in various program sectors and a Technical Advisor for M&E. The ACR-MQ oversee the functions of Directors of Finance, Material Resources, Human Resources and Administration. At the state level, the SR manages the DAP and non-DAP programs with the assistance of a team of 3-7 Partner Support Officers (PSOsF 17), Finance Officer, Administrative Officer including staff assigned to monitor the use of DAP food resources. PSOs coordinate, guide and support their respective partners. PSOs also make monitoring and support visits to the program sites in their respective partner areas (See the CRS/India organigram in the annex). The change from the zonal structure of CRS to the state structure has advantages, many of which were noted in the MTR and the FE. CRS and partner staff believe that programs are more meaningful because CRS is located closer to the communities to facilitate frequent monitoring. Partners noted that although CRS tried to be responsive under the zonal structure, under the new structure CRS support is (more) readily available, personal interaction has increased and relationships are strengthened. Each state level program can grow and evolve as per the specific needs of the state; location specific strategies are being developed. Within the CRS State Offices, there has been a change from sector specific project managers to Partner Support Officers (PSO) after restructuring to facilitate communication with CPs. In the new system, each PSO is responsible for a particular number of CPs regardless of the sector. This means that the CRS staff has to be knowledgeable about many sectors and the team has to work closely together. In the State Offices visited by the sector consultants, it was evident that the staff members work closely together. However, some evaluators noted that as a result of this change in structure, there has been a reduction in the availability of technical personnel since these individuals - while still part of the structure- now have additional responsibilities outside of their sector expertise or a different position. Transferring technical staff from the zonal offices in the old structure to management positions or multi-sector PSO positions in the new structure was carried out intentionally to permit staff to have a broader appreciation for the work of implementing partners and for the management of the program in general, and to enhance possibilities of integration. PSOs are shifted within a State Office with the intention of bringing fresh ideas to different project sites. There is also cross-fertilization between State Offices and State Offices frequently call upon staff from other offices for technical advice, especially in the agriculture sector. While this approach should be encouraged, cross-fertilization and exchange of best practices between State Offices could be increased since evaluators found that there are many good models and excellent tools which are not always being shared. As noted in the MTR, CRS still has lean staff sizes. A few staff positions are vacant or new due to some recent attrition. Moreover, State Offices and the National PQ Team are still adjusting to the new structure which has an impact on implementation and monitoring. The risk is that new staff will take time to learn the new program and monitoring may be routine. 17 PSO is a CRS-specific title equivalent to a Program or Project Manager in other organizations. PSOs are responsible for all projects implemented by the partners they are assigned to support, both DAP and non-DAP related. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 42 Implementing Partners CRS has been implementing DAP II in partnership with a wide and strong network of local organizations including both Catholic Diocesan Social Service partners and non-church partners. CRS State Offices provide managerial support, technical guidance, and advice for preparing annual plans, implementing project activities, and developing human resources within the context of the DAP. PSOs help partners to prepare project reports and evaluate outcomes. CRS also helps CPs to undertake a variety of non-scheduled project activities to enhance their overall capacity, such as writing project proposals to diversify funding from CRS and other national or international donors. CRS provides substantial support to their implementing partners: their level of involvement in project implementation is quite high, as observed during the final evaluation, and monitoring was regular. CRS facilitates the work of the implementing partners just as these organizations facilitate the work of the village institutions. There is a close relationship between CRS and implementing partners, at all levels of management, and between CRS and the village level institutions. Aside from the formal meetings that take place for reporting purposes, informal communication takes place on a regular basis. At the partner level, CP directors manage a variety of programs with the assistance of sector coordinators. Sector coordinators provide guidance and support to OPs in program management and implementation. Sector coordinators also make monitoring and support visits to OP areas. At the grassroots level OPs implement the program with a staff of supervisors and village level workers who work hand-in-hand with community structures. Most implementing partners (CPs and OPs) have adequate capabilities (administrative, infrastructure and managerial) for handling food for distribution, as well as for implementing sector specific activities. For example, in education many partners are running their own schools and have a solid knowledge about and relationships with other neighboring schools which facilitates referral of ECDC/OCF children for further schooling. In addition, a majority of the partners, especially CPs, manage development activities supported by other international and domestic donors. Therefore, support from the CRS is supplementary as well as complementary, enabling greater social and economic changes in these areas. This strength of partners has implications for sustainability. Capacity Increasing the capacity of partner and community level staff through training and other on￾site learning is one of the primary approaches to improving the quality of services in DAP II. One of the major strengths of the program is the investment made by CRS to build the capacity of local partners and village level institutions, as well as their own staff. CRS has developed training programs for all partners covering technical and management subjects and skill building. Aside from workshops and formal training events, CRS has used innovative techniques to assure on-site learning. For example, over the life of the project CRS and CP staff organized 152 exposure visits in agriculture and 84 for health. In the agriculture program, CRS has developed a high degree of technical capacity among CRS and CP/OP staff. These staff demonstrate strong abilities to manage watershed development projects and their ability to innovate and adapt technical interventions to location specifications. DAP II has trained 41,808 farmers and 2,905 AEWs over the life of the project. In the field of education, a majority of partners send teachers for training at government recognized institutions such as the District Institute of Training (DIETS) since Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 43 teachers in the CRS supported program use the GoI curriculum. Since the beginning of DAP II CRS supported the training of more than 5,692 ECDC teachers, 8,975 primary school teachers, 922 bridge course teachers and 1,320 school teachers. Over the life of the project CRS trained 9,919 VHW in health and nutrition, MIS, Community Mobilization and 5,592 TBAs and 1,128 RMPS under the SMCS program. CRS The CRS has well developed National & State Offices with sufficient infrastructure to handle the DAP II program in a comfortable manner. The State Representative and PSOs have adequate skills in program planning and implementation. Although learning has been a priority, more training may still be required to bring new CRS staff up to speed. In 2005, CRS completed a written inventory of all staff core competencies and areas of weakness which is being used as a basis to upgrade skills and carry out professional development to improve performance. CRS has already created small booklets documenting success stories, such as those found in the Rajasthan State Office. Nevertheless the evaluation team found that CRS needs to extend the sharing of these documents with like-minded organizations at the State and National level so that other institutions learn from experiences. By increasing its interaction with peer agencies working in the same sector CRS will also learn about the innovative work of other NGOs and be better informed about current National programs and policies, and funding or resource opportunities. Partner Capacity Many of the CPs and OPs who are implementing partners in DAP II were also working with CRS in DAP I program. The CPs have built solid linkages with the local government officials in the specific sectors, and CRS and the CPs have forged relationships with the appropriate government departments and concretely involved them in project activities by inviting them to co-facilitate trainings and attend community level activities regularly (e.g. health days). CPs are recognized for their strength as trainers: partners were involved in teacher training, materials development, community mobilization and micro-planning workshops. For example, some of the CRS partners now serve as resource agencies to other partners for teacher training. CPs could improve their capacity by strengthening their technical competency, especially in public health. SMCS Coordinators are incredibly committed, but their technical capacities are not sufficient for their expected responsibilities. It is recognized that in some remote places finding personnel with such qualifications may be difficult. While most CPs are trained as trainers, CRS and CPs can improve their training program by adding a learning needs assessment tool, pre and post- tests, and training evaluations to provide feedback on the quality of the training itself and to understand what participants have learned. Sector consultants were encouraged to see that OPs who had no previous experience in implementing natural resource management projects, are today as capable as those OPs who have had years of experience. For example, one CP with a fair knowledge of the technical aspects of watershed management has done a remarkable job building the capacity of OP Umergaon in Bastar district, Chhattisgarh. The CP demonstrated that village level institutions are sustainable, even in this area where people still follow a very traditional way of life. Such successes are due to the commitment of CRS staff and to the cooperation extended by CPs in training the OP staff and facilitating the implementation process. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 44 CPs have also demonstrated their ability to innovate. For example, the CP GRAVIS in Rajasthan has developed silt traps tankasF 18 to ensure cleaner drinking water and spillways for khadinsF 19, and has adapted pastureland development models. Based on the observations from the sector consultants, CPs should continue to call upon the appropriate government officials to facilitate learning events, especially for VHW/TBA training, in which the Block Medical Officer of Health should be involved to increase capacity. Including local government officials not only builds relationships and strengthens linkages but also ensures that project activities, such as training of TBAs, are in line with GoI policies. Community In all three sectors community-level structures have proven to be a source of strength for project implementation. Village based committees have successfully mobilized communities around health, agriculture and education and have been a motivating force encouraging villagers to access services offered by both the project and the local government. In some instances, community-based institutions themselves have advocated for certain services from the local government, such as lobbying for the construction of a water source in the village. Furthermore, these institutions have monitored the execution of the activities themselves (e.g. health days or construction of water harvesting system) and defined rules and regulations outlining community participation and access to services available. The institutions, trained by the project, set up internal governance procedures and elect a President and officers. The committees meet monthly and establish funds into which voluntary contributions from the population are placed in order to sustain future activities. Such institutions should be further strengthened and motivated to continue their important role and harnessed for future programming. Although the village level committees exist for education and take responsibility for cooking the food and monitoring the follow-up children at risk for drop out, little data is available to chart the progress and successes of these structures. The VECs are mandated by the constitution but do not appear to have the same extensive role in the project as the WC and VHC in the other sectors. There was little evidence of traditional Parent-Teacher Associations at the school level during visits or in the documentation for the program. The role of the VEC needs to be redefined as mentioned in the Education Chapter. SHGs have been the most successful community-based group. These institutions are created to facilitate savings among marginalized women, but have grown to become strong civil society organizations and proven agents of change. In many cases SHGs are active in social issues facing their community, and in certain cases SHGs members have been elected as local government representatives. While SHGs existed before the DAP, CRS has expanded the number of SHGs with private funds as part of DAP II. CPs and OPs have trained the SHGs to establish internal regulations and manage their finances. ith these successes, SHGs should be seen as another successful community-level strategy to expand in future projects. Evaluators noted that since the MTR, SHGs have become more integrated with the sector activities. 18 TANKAS:Traditional underground water harvesting and storage tank 19 KHADINS: earthen embankment built across the slope which conserves the maximum possible rain water runoff within the agricultural field. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 45 Although these village level structures are quite active, their links with the PRIs could not be easily detected by the evaluation team, except in the Agriculture sector due to land and water related interventions. Links with PRIs are extremely important to guarantee that the program is accepted by locally elected officials and to ensure sustainability. Village based extension agents in health and agriculture have done an excellent job of creating awareness among the community and have contributed to expanding knowledge by providing technical assistance especially in watershed management. Management Information Systems Observations While CRS has hired a Technical Advisor for Monitoring and Evaluation under the Title II program at the national level, there is no one in the State Offices who is directly responsible for the database. Furthermore, although CRS has installed database software for program monitoring, and developed and installed a food and fund tracking software based on the needs expressed by program staff, some offices have encountered technical difficulties. Nevertheless systematic records were observed at CP and CRS offices; and registers are available at the community, although some are not filled out completely. Although CP monitoring was good due to their long experience, improvements could be made to standardize monitoring and feedback especially in health and education. In some places where monitoring tools exist, they are not being used consistently. Some CPs commented that at times monitoring visits were not planned in a timely manner. Training was provided to CRS and CP staff, with accompanying manuals, in all sectors and there is plethora of good data being generated by the village, partner and CRS level. However it is not clear if the MIS is being used to its full extent to make programmatic decisions. Feedback is not provided consistently to partners and communities in all State Offices, although information is shared informally. It is unclear if information from the MIS is used when providing feedback. Monitoring visits are frequent but there is no information to determine the quality of the monitoring visit. In some interviews, qualitative data revealed that staff believe that at times monitoring visits are very routine. If staff had fewer partners and project sites to monitor they may have more time to spend during monitoring visits. Based on the FE, it appears as if qualitative aspects of the education program are not systematically monitored in the MIS. Some members of the evaluation team felt that there was a very strong emphasis on collecting quantitative information and that data collected by the MIS may not help sector specialists and supervisors to make quality improvements. Quality improvement indictors should be developed for all three sectors, especially for education to evaluate the achievement levels of children. In addition, schools should be tracking survival rates, attendance trends and attrition – and use this data for decision￾making. Presently these rates are only compiled at the CP and CRS level for managers. Monitoring In FY ’05, CRS staff visited 311 villages in the health program, and spent about two days visiting each village. CP staff visited 2,860 villages, out of 3557, in the health program. For education, CRS staff visited 468 villages and CP staff visited 3,631 villages. CRS visited 507 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 46 villages in Agriculture (or at least two visits to each watershed) and CPs visited 2,362 villages. Nevertheless, through the evaluation it has been noted by the sector consultants that monitoring should be less routine and focus more on the qualitative aspects of the program and its activities. CRS has hired technical staff and put in place appropriate software to collect information on program, commodity and financial activity. Staff at all levels, including community extension workers, have been trained in the MIS. Data generated from this system is used to complete monthly, quarterly and annual results reports according to donor requirements. Based on the recommendations of the MTR, and lessons from the field on self reporting, CRS initiated a process to further simplify and refine the MIS in FY2000. The MIS now has two components: self-reports by stakeholders and the information gathered by the supervisors through periodic supervision and monitoring visits. In the latter system, both CRS program staff and Coordinating Partner (CP) level program staff directly collect information on key monitoring indicators during visits. Gaps in the data: Data collection/Indicators unrealistic: Watershed Committees have not achieved the target of 100% for rules and regulations on water distribution. This is a very difficult target to achieve, and it is felt that the target was unrealistic since it is difficult for communities to sit together and make regulations about water use when they do not actually see any water. Furthermore, this has been seen as a difficult achievement in other watersheds supported by other organizations in India. Data collection/Indicators not appropriate The indicator developed for water security, number of weeks wells retained water, is not appropriate and if used, should be analyzed in conjunction with other water security indicators. Because CRS did not construct these wells, there are too many variables which could affect data collection and the results. Water catchment systems created under DAP II are storing harvested water throughout long, dry periods, but there is no data collected to demonstrate this increase in availability. Data Collection/Indicators not sufficient In many instances, results achieved under DAP II are not captured because there is no indicator to gather the information. For example, in locations where water is being retained in surface water bodies for a longer period, farmers have added fish farming and use the water for irrigation. However there is no mechanism to capture these additional watershed benefits. In health, there is not data collected to confirm if mothers consume IFA tablets or not. Many interviewees stated that more children are going to school; this information needs to be quantified. MTR Follow Up A Mid-Term Review of DAP II was conducted from July to November 2004. The overall conclusion of the MTR was that “CRS/India has done extremely well what the 2002-06 DAP required of it during the 2002-2004 period reviewed in this MTR report. Virtually all indicator targets have been achieved in a timely manner and many have been exceeded.”F 20 However, the MTR raised several key issues related to the overall impact of DAP II on food 20 MTR Report, p. 106 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 47 security. In May 2005, CRS finalized a detailed plan to address the key MTR findings. Each CRS State Office has established a plan to address the recommendations. Despite the fact that the MTR was completed 18 months ago, and the plan to respond to the MTR recommendations was finalized in May 2005, many recommendations have been completed especially in health and agriculture; recommendations have not yet been followed up in the education program because the recommendations were less specific and were of a long-term nature. Those recommendations which suggested additional analyses or studies are being completed now during the development of the next phase project. Furthermore, each SO has a preliminary phase out plan and is developing State Office-specific strategies which was also recommended by the MTR. Furthermore, CRS has made efforts to increase staffing and strengthen capacity of personnel (please see Capacity section to review training accomplishments). Other recommendations include integrating the watershed, health and education activities which is being done on a pilot basis in several communities. Moreover in FY ’04 CRS developed a phase-out strategy (which has been modified several times) and started to investigate possibilities to diversify its funding base and networked with other development actors in India to increase its visibility and forge linkages for greater collaboration. For example, in April 2006 CRS responded to an RFA from PATH to implement Maternal and Newborn Care services in the UP and in May submitted a Concept Paper to the Gates Foundation for quality education. Please see the sector reports for specific example of how each sector has responded to the MTR recommendations. Sustainability DAP II was designed having considered issues of sustainability at the community and partner level. Both the health program and the agriculture program have exit strategies defined as an element of project implementation so that villages graduate from the program and discontinue receiving resources once targets are met. Capacity building is the cornerstone of the DAP II strategy to improve knowledge and practices of household members and increase skills of community- based structures to continue to carry out village based activities on a self￾motivated, voluntary basis without the continuing inputs of the program. Institutional development also assists implementing partners so that they can become strong development actors in India. DAP II project staff concentrate on training staff at the partner and community level to sustain knowledge. Program implementation ultimately is focused on sustaining the outcomes and results by changing behavior and improving practices in health, education and agriculture. Implementation of program activities supported under DAP II can be further sustained by linking community- based activities with government facilities providing health, education and agricultural extension services. Government programs also provide technical assistance and other support in some cases. These linkages will provide an opportunity to continue to maintain gains made under DAP II in knowledge and practices and motivate communities to uphold volunteer efforts they made under DAP II. Food resources may be sustained only in some sectors, and not in all communities. In some cases, additional cash resources may be available to retain village level staff from either government schemes or private donors. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 48 Implementing partners at the OP level are working to establish relationships with national programs and other donors to obtain new sources of funding. The vast majority of CPs already has funding in addition to Title II support, some of which is from CRS. To date CRS has held several strategic planning sessions to consider issues of sustainability in face of the pending phase out of Title II resources from India to further study issues which may not have been considered during the design phase of DAP II. CRS has explored the possibility of building formal linkages with government programs through advocacy; documenting models and best practices and diversifying its funding base to support them; as well as a geographic consolidation and prioritization of program activities to help naturally reduce expenditures as resources from Title II decline. In line with this strategic reflection, CRS has developed a preliminary phase out strategy, which includes many of these themes, and has worked with each of the CRS State Offices to develop draft phase over/phase out plans that include specific actions to take during the next five year period. Although Title II resources have supported CRS food security programs for many years, community based activities began only under DAP I in 1997. Furthermore, DAP II represents a departure from many activities in DAP I and an overall improvement in strategy and implementation in all three sectors. Strategies deployed under DAP II have proven effective as evidenced by the final evaluation. However given more time, CRS could extend these successes to more households and consolidate gains made over the last five years. The evaluation team therefore is recommending that the program be supported for an additional five years, which would imply a total of ten years of support to these initiatives begun under DAP II. The additional food resources will allow community-based activities and institutions such as primary schools, time to build solid linkages with government programs and secure the resources necessary from these schemes to ensure continuity in program activities. For the future five years, sector consultants have recommended that food resources should be provided for five years in education and humanitarian assistance programs, and three more years in health and agriculture. However if five years of food distribution could not be made available for education, a three year minimum would be acceptable. While plans to link current activities in all three sectors with government schemes is the suggested course of action to sustain results made under DAP II so investments are not lost, CRS must make significant efforts also to prepare villages and partners for the end of the Title II program which has been in India for many years. Village based institutions must be well prepared to sustain the program and they should have a clear idea of what to expect over the next few years during the phase out period, and thereafter. Experience has shown that in villages which graduated such institutions did not maintain project activities because they were ill prepared and eventually dissolved. Village institutions will have more support from the local government and greater access to government schemes if they are formally recognized by the PRI. Beneficiary Contributions: As per USG regulations governing use of Title II resources, beneficiary contributions which are generated from the sale of empty containers and the cash contributions made by participating households which receive food must be spent by the end of the project. Currently these contributions are used to cover food transportation (CP to OP) and administrative costs at the partner level. CRS keeps track of the balances on hand at the partner level on a regular basis; these balances are available on file at CRS/India. Presently, there are small balances on hand since the majority of these funds have been used to transport Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 49 food aid. In addition, CRS/India has developed very strict guidelines for partners as to the use of these funds, including restrictions on disbursements. These guidelines are also available from CRS/India. Nevertheless, it is highly recommended that CRS/India put a process in place for the next phase project in which partners begin to reconcile balances of beneficiary contributions, and estimate the level of contributions that may be on hand 12 months prior to the end of food distribution. With this estimate in hand, partners should develop small proposals to support community-based and community identified priorities for development activities. These activities should be related directly to the objectives of DAP II, namely food security, and should also somehow be aligned with the project’s interventions. The activities also should be supported in those villages which participated in DAP II. Simultaneously, CRS/India internal audit staff should also visit the 60+ CPs to verify any on-hand balances. In order to facilitate this process, CRS should develop project proposal guidelines, and request proposals from those partners with balances on hand. CRS should inform partners early of these restrictions, and of the process for the call for proposals, along with definite deadlines for the proposal submission as well as the due date for financial reconciliation. The call for proposals should be made well enough in advance so that partners have at least 6 months to implement the project and 3 months to account for the expenditures with a financial report. The receipts should be due at least 90 days before the end of the fiscal year, in case there are any questions related to the financial reports. It is strongly recommended that the projects be simple, one-off projects such as a well. Lastly, there is no possible way to complete these activities in the last year of the next phase project if there is still a collection of beneficiary contributions, since this would generate more funds which would remain as on-hand balances at the end of the project. It is therefore suggested that CRS waive the beneficiary contributions in the last year of food distribution, and support these costs with alternative funding. In this way partners will focus on spending down any existing balances through the process identified above, and not continue to collect funds which could not be used during the life of the project. PHASE -OUT Priorities: CRS should plan to reinforce community structures. Community structures should understand what the end of the project means and a discussion should be facilitated in which communities make decisions about how they will continue the activities and document them in an action plan. Already funds are collected by health and watershed committees which should help greatly. CRS should additionally work intensively with CPs and OPs to strengthen their capacity. An assessment should be done to determine which CPs already have skills and access to other donor funds with which they can continue activities; CRS should plan to help those partners through skill building and link them with international and national donors. Strategies CRS should take during the Phase out: CRS should make some strategic decisions about which activities it wants to continue to support; and should focus on certain parts of the country in order to reduce the scope of the program. CRS State Offices should develop definitive criteria and make these decisions in the last quarter of FY 06. CRS should remain working in underserved areas where partners are Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 50 strong; or in underserved areas where there is a partner with potential. Weak partners, and partners with limited potential, or partners with a strong funding base could be spun off. NGOs which will not continue with CRS in the future, and have wide access to other resources should be spun off immediately, so CRS has time and resources to focus on other partners who may not have the same level of access. CRS should use these strong or “Tier I” partners to mentor other partners who may be Tier II. Tier III or Tier IV partners whose capacity is extremely low or whose vision may not be the same as should be evaluated and CRS should decide, through a prioritization process using pre-established criteria, which partners should be phased out. Next Steps: 1. CRS must ensure that target communities have made formal linkages with local government departments so that the same services that were being provided under DAP II continue. CRS and partners should inventory every community covered by DAP II and determine which communities may still need help from CRS or the CP/OP to facilitate this linkage. CRS and/or the implementing partners should not simply provide information to the community institution about the program, but they should actually accompany the committee to the government office and complete forms if necessary. 2. As part of this inventory, CRS and partners need to note which communities from health and agriculture will be graduated by the end of the PHASE OUT and which will not. Graduating villages must have linkages with government schemes; however non￾graduating communities will require greater amounts of additional resources and accompaniment if the same model of implementation is to be followed. 3. CRS should not add any new communities or activities during the phase out period, but should concentrate on making the linkages as described above and further strengthening the community-level institutions so that they are self- reliant and empowered to the greatest extent possible by the end of the next phase. 4. CRS should focus on expanding best practices across all three sectors and discontinue any pilot activities which are unsustainable or unsuccessful activities. Any activities which can already be taken up by government services (ICDS, Midday Meals) should be phased over to those services by the end of DAP II, or in the first quarter of the next project. The scope of activities in the next phase project must be very focused. 5. Those communities slated to graduate from the program under health and agriculture should be allowed to do so according to the original plan, and to the extent possible CRS should try to accelerate this timeline. Additionally CRS should try to graduate more communities than originally planned, if possible. One immediate course of action would be to determine if there is already any overlap or duplication in CRS targeted villages with the ICDS program in the case of health/education, or with the mid-day meals scheme in schools. Because resources are limited in the next phase of the program, these communities which may be already participating in the ICDS for example, should be phased out without delay. a. Evaluate communities during the inventory process proposed in (1) above; communities which may have participated in the program for fewer months, but which are stronger, could qualify to graduate early. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 51 6. CRS should shore up program quality and expand progress made under DAP II to attain a greater coverage in the target community. By enrolling more households in the program and expanding mobilization CRS can contact hard to reach families. Health: o Continue to improve access to safe motherhood and child survival services by reinforcing the existing link with the local health department. o Continue health education and mobilization activities. Phase existing activities over to the ICDS program, if the program is active in the SMCS village. ICDS is modeled on the same approach and offers many of the same services to communities. Notwithstanding gaps with ICDS coverage and quality, CRS can continue exploring possibilities of phase over the SMCS program participants to the ICDS program. Furthermore, the NRHM scheme has planned to hire an ASHA and establish a Health and Sanitation committee in each community. By reinforcing linkages with the NRHM and establishing a formal relationship with the PRI, CRS and partners could facilitate that the VHW under SMCS is chosen as the ASHA and the VHC becomes the Health and Sanitation committee. While in some cases this is not possible since other factors may influence the selection especially of the ASHA, CRS should try this approach where feasible. o Maintain the VHW as a motivator and communicator to local families, as this has been seen as a key best practice, if there is no ICDS program. However the VHW can only be maintained if she continues to receive her salary, as this is not a volunteer position. Information gathered during the FE demonstrates that if the VHW is not paid she will not continue her position, and the linkage with the local health facilities is lost. Agriculture Villages that are graduating should be linked with government extension services for continued technical support. CRS should ensure that the Watershed Committees are strongly rooted, have clearly defined rules and procedures for use of water harvested and well prepared for the end of the project. Although each of the watershed projects have funds created from cash contributions, CRS should make efforts to strengthen these WCs and ensure that they have a means to continue collecting funds from voluntary contributions so that assets can be maintained. CRS should ensure that communities are aware that they will not receive any FFW for maintenance work and neither will they be paid any cash, and that they are aware that after the completion of the project, they will have to contribute voluntary labor. CRS should take more time to strengthen WCs and to complete watershed feasibility studies and construct water harvesting structures required. CRS should make efforts to learn more about the NREGS program as it scales up and become operational and link these communities with the NREGS program. At this time, the parameters of the program are unclear: although funds have been approved for 200 districts, criteria and procedures to access these funds are not defined. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 52 Education: CRS needs to sustain food support and quality education which is presently available so children from poor families continue attending and graduating from school. In that case, Title II food should continue in private schools for some time since Mid Day meals has not been designed (yet) to cater to these schools. Current the MDM program targets government and government supported schools. If the Title II food is not continued, the private cost of education will increase and children from marginalized communities will not be able to afford higher school fees. Therefore Title II food should continue until the GoI covers private, poor-resourced schools or if not, for as long as possible. Coupled with this, CRS and USAID should lobby the GoI to cover these private schools which serve children from marginalized families since otherwise these children would not have access to education. CRS will need to develop an exit strategy for the education program. One clear means to sustain the progress made in education is to link the communities with government schools that exist nearby, which have access to the mid-day meals program. The first step would be to target mothers as they have been seen to be the key decision-maker or influence whether and where the child goes to school. Through intense mobilization and counseling the program will need to build the mother’s confidence in the government school. It is suggested that the program set up meetings between the village education committee or representative group of parents and the government school so that families currently sending their children to CRS schools will feel more comfortable and will make the shift more easily when Title II end. However this strategy will take time. At the same time, CRS and partners should work closely with the government schools to upgrade their quality. CRS should also sustain the residential institutions that serve children from remote locations. For this, DAP II should continue to support the OCF programs based on these criteria. CRS should develop a sustainability strategy with guidelines, criteria determining which schools will be continue to be supported, and which ones will be phased out immediately. The other option is to seek alternative resources to support the CRS-supported school if there is no government school in the vicinity. Finding alternative resources may prove to be challenging, although CRS and its partners should lobby the government to provide Mid-day Meal at CRS-supported schools in locations where there are no other education services and where CRS partners are providing education to high percentage SC/ST/OBC populations. In the case of Bridge schools, children will be mainstreamed to government schools by design of the intervention. However CRS will need to identify alternative resources to support the Bridge schools in preparation for the end of Title II support. Currently many of these schools receive support from CRS private funds, but a large number of schools are being supported by the National Child Labor Program. This collaboration should be seen as a model for other education activities, and should be expanded to reach all 106 Bridge Schools supported by CRS under DAP II. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 53 Recommendations 1. Redesign the MIS so that it is user-friendly and seen as a valuable tool for managers. Design new indicators to capture results and qualitative aspects. 2. Reinforce the training program in all sectors so that competency is measured. 3. Empower community based institutions so that they can operate village based activities which create demand for services. Work with communities so that they demand high quality services in health, education and agriculture from the government. Realign roles of village based institutions and staff. 4. Realign the role of CRS PSOs to become linking agents, helping CP/OPs and community institutions to find new resources to sustain DAP Activities. Emphasis should be on linking partners at all levels with GoI schemes. 5. Fill vacancies in the PQ team and ensure that it is assuming its role in the new structure. 6. Document success strategies and innovations, and take them to scale but do not start any new activities, locations or initiatives. Coverage should extend to the entire village – increase coverage in the next phase. Share best practices between SOs. In the future, new communities should not be added to a program until significant progress has been made on project indicators and at least 80% coverage has been attained. 7. Prioritize, consolidate and phase out any programs that are already “resource-rich”; seek to reduce the spread of the program to improve monitoring. 8. Spin off well-endowed NGO partners and develop a mentoring programming with resource-poor NGOs to increase their funding levels. 9. Increase participation of women in decision-making, so that they are not simply present in committees but that they actually contribute. Ensure that village based institutions such as Watershed Committees are not dominated by a few educated villagers resulting in unequal decision-making. 10. Coverage in many villages has been an issue. CRS should examine the relationship between limited quantities of food and how this restricts the agency’s ability to reach greater community enrollment or number of participants in programs such as health, in particular, and education. Concluding Remarks DAP II has contributed enormously to improving food security in the project target areas where more than 80% of participants are from the most disadvantaged communities living in India. Agriculture activities promoting a watershed approach have led to a decrease in migration and an increase in household income, as reported by project participants and as evidenced by the final evaluation survey results. The watershed project has been highly successful in Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 54 increasing factors which lead to increased productivity including augmenting the acreage of land under cultivation, increasing double cropping and improving crop diversity. Participants are using higher yield agricultural inputs such as seeds and fertilizers, and more are seeking advice from extension agents and building linkages to resources outside the project for the future which will sustain impact realized under DAP II. The project’s success can be attributed to a strong element of grassroots mobilization, greater community cohesion and strength of the Water Committees through the pre-watershed phase, intensive monitoring by dedicated technical staff at CRS and implementing partners, technical support from the AEWs and the watershed approach itself. The challenge facing the education program is how to sustain the provision of services to marginalized populations. Schools supported by CRS under DAP II will continue to function since they will collect school fees to cover their expenses. However, children from low￾income households will most likely not be able to afford such fees. If the private schools are not able to offer subsidies or scholarships from NGOs, foundations or government schemes, these children have no choice but to attend public school or drop out if there are no schools in their area. In this light CRS and its partners should begin to prepare families to accept the government schools. Drawing on past experiences, CRS and its partners should help the children transition to the closest government school. ECDCs could easily be absorbed by government schools or community programs and then closed: during the final evaluation, the team noted that this was already happening in several cases. Where government schools do not exist within a reasonable distance, CRS and its partners should lobby the GoI to support the CRS school. Overall, the SMCS program achieved results by changing knowledge and practices in almost all areas of Safe Motherhood and Child Survival. One of the most consistent improvements seen across all areas of care was the preference of women to seek care/consultation at a health facility as opposed to seeking services from non-medical professionals outside the hospital or health center. Although the gains made were small, in terms of percentage points, they are noteworthy since these are difficulty areas in which to change behavior. Moreover, the change noted from the baseline signifies that the strategies used to raise health awareness and increase access to services by forming strong links with the government health services are working. CRS should try to expand the coverage and increase targets in the remaining years under Title II support to consolidate and sustain the change in behavior. While plans to link current activities in all three sectors with government schemes is the suggested course of action to sustain results made under DAP II so investments are not lost, CRS must make significant efforts also to prepare villages and partners for the end of the Title II program which has been in India for many years. Village based institutions must be well prepared to sustain the program and they should have a clear idea of what to expect over the next few years during the phase out period, and thereafter. Experience has shown that in those DAP villages that have already graduated, such institutions did not maintain project activities because they were ill prepared and eventually dissolved. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 55 Annex I: Evaluation Scope of Work Scope of Work for Final Evaluation of TITLE II assisted CRS/India programs January-May, 2006 Revised 5 February 2006 I. Background A. Catholic Relief Services/India Program Catholic Relief Services (CRS) is an international relief and development agency serving the poor in over 100 countries through programs in emergency relief, HIV/AIDS, health, agriculture, education, micro finance, peace building, and safety net programming. The relief and developmental interventions across the countries are co-ordinated through the agency’s head quarters located at Baltimore, USA. CRS officially started its food-aid assistance to India with signing of the Indo-US Agreement in 1951. Initially, the Title II food resources from the USG were used to support a "family feeding program". Over the years this program evolved to direct interventions in mother and child health, education, agriculture and humanitarian assistance. Since 1997 Title II programs have been planned and implemented in five year cycles known as Development Assistance Programs (DAPs). During DAP I (FY 1997-01) CRS/India concentrated its efforts on the transition from a development program focussing on needy individuals to a community based approach that actively engaged CRS partners, village level groups, and program participants to develop and implement sustainable development activities at the grassroots level. In DAP II (FY 2002-06), community based approaches initiated in DAP I are being strengthened. Until October 2003, CRS/India co-ordinated its programs through a country office in Delhi, four Zonal offices at Lucknow, Calcutta, Mumbai and Hyderabad, and select State Offices at Bhubaneshwar, Bhopal, Guwahati, Jaipur, Raipur and Ranchi. In order to improve and strengthen partnership and to facilitate decision making, this three tier structure was replaced by a two-tier structure. A national office in Delhi (along with staff working at national level from the Hyderabad office) coordinates CRS/India’s activities through 12 State Offices. The State Offices work directly with more than 60 larger NGOs called Coordinating Partners (CPs) in 23F 21 states (including the seven north-eastern states) and the union territories of Andaman and Nicobar Islands, Chandigarh, Dadra and Nagar Haveli and Pondicherry. CPs in turn co-ordinate with around 2500 grass root level NGOs called Operating Partners (OPs). With respect to the DAP, these OPs work with around 7000 villages /educational institutions, most of which are located in remote areas of the country. B. DAP II Program Framework DAP II, like all CRS/India programs, targets the most vulnerable groups (SC/ST/OBC). Health sector program activities target children under 3 years of age, along with pregnant and lactating mothers. The DAP agriculture program targets small and marginal farmers and the education program targets disadvantaged children, especially girls, in the primary school age groups. The humanitarian assistance program targets victims of calamities, the destitute, 21 Andhra Pradesh, Assam, Arunachal Pradesh, Bihar, Chhattisgarh, Gujarat, Himachal Pradesh, Jharkhand, Karnataka, Kerala, Madhya Pradesh, Maharashtra, Manipur, Meghalay, Mizoram, Nagaland, Orissa, Rajasthan, Tmil Nadu, Tripura, Uttar Pradesh, Uttaranchal and West Bengal. Please note that DAP activities do not cover Tamil Nadu, Karnataka, Kerala, Pondicherry and Andaman and Nicobar Islands Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 56 orphaned, sick and dying, especially children (please see DAP II document for program description). The overall goal and strategic objectives of the CRS/India 2002-2006 DAP are: Goal: To improve Food Security through the participation of India’s most vulnerable groups in sustainable development. This goal will be achieved through the following objectives: Health 1. Improve health of 240,000 pregnant/lactating women and children aged 0-3 years Sub objective: 1.1 Ensure safe and healthy pregnancies for 90,000 women and improve nutritional status of 150,000 children aged 0-3 years Agriculture 2. Increase agriculture productivity of 200 farming communities during the five year DAP period Sub objective: 2.1 Increase multiple cropping by improving water security for agriculture Education 3. Increase opportunities for and participation of 350,075 disadvantaged children (SC/ST/OBC) annually, especially girls, in quality primary education Sub objectives: 3.1 Ensure access to primary education for 350,075 disadvantaged children (SC/ST/OBC) annually, especially girls. 3.2 Improve educational quality in 3,535 program schools Humanitarian Assistance 4. To provide a safety net to victims of calamities, the destitute, orphaned, sick and dying, especially children Sub objectives: 4.1 To provide short term emergency food supplements. 4.2 To supplement food intake of orphans, the disabled, destitute and dying poor including HIV/AIDS infected persons and HIV/AIDS affected children, in humanitarian care. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 57 The Title II assisted DAP II programs of CRS/India reach 967,795 program participants annually. As per the approved DAP submission, DAP II was expected to conclude by December 31, 2006. However, in April 2005 CRS and USAID mutually agreed to realign the DAP with fiscal year cycle. Accordingly the current DAP will conclude by September 30, 2006. In the interest of program quality and in accordance with DAP requirements, CRS wishes to conduct an external final evaluation of the Title II assisted DAP II program during January – mid May, 2006. Currently, CRS/India is developing the next phase of a Title II development program cycle (a phase out programF 22) which will begin in October 2006. As per the USAID requirements the DAP final evaluation must be submitted before the next program proposal is formally reviewed. The proposal review is expected to start by the end of May 2006. Hence, it is imperative that the final evaluation is completed by mid-May, 2006. C. DAP II Program Management Overall, the CRS/India DAP II is implemented in 20 states including the seven North Eastern states and the union territories of Chandigarh and Dadra and Nagar Haveli. However, the number of states covered for individual sectors vary. Details of states covered in the individual sectors are provided in Annexure I. As with all CRS/India programs, a team comprising of both international and national staff manages the CRS/India DAP. A Country Representative (CR) leads this team with the assistance of two Deputy Country Representatives (DCRs) based at Delhi and Hyderabad, an Assistant Country Representative for Management Quality (MQ) and the department director of Internal Audit. The Deputy Country Representatives are assisted by two Assistant Country Representatives (ACRs) for Program Quality (PQ) and 12 State Representatives (SRs). The ACRs-PQ oversee a team of National Coordinators in various program sectors and a Technical Advisor for M&E. The ACR-MQ oversees the functions of Directors of Finance, Material Resources, Human Resources and Administration. At the state level, the SR manages the DAP and non-DAP programs with the assistance of a team of 4-5 Partner Support Officers (PSOsF 23), Finance Officer, Administrative Officer and other support staff, including staff assigned to monitor the use of DAP food resources. Each of the PSOs coordinate, guide and support their respective partners. PSOs also make monitoring and support visits to the program sites in their respective partner areas. At the partner level, CP directors manage programs with the assistance of sector coordinators. Sector coordinators provide guidance and support to OPs in program management and implementation. Sector coordinators also make monitoring and support visits to OP areas. At the grassroots level OPs implement the program with the assistance of supervisors, village level workers and the community. 22 Previously known as DAP 23 PSO is a CRS-specific title equivalent to a Program or Project Manager in other organizations. PSOs are responsible for all projects implemented by the partners they are assigned to support, both DAP and non-DAP related. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 58 D. DAP II Resource Context Resources for the DAP programs include Title II food and financial resources from both USG (202e) as well as CRS. Key resource constraints faced during the DAP II period are detailed in this section. Title II food resources planned for the DAP II period included Bulgur, Corn Soya Blend (CSB) and Oil. Food resources are used as incentives for participation in the health, education, and agriculture programs. The ration rate for each sector is defined in the DAP project text. However, in November 2002, due to issues surrounding Genetically Modified foods, CSB was not only dropped from the food resources but also not replaced with any other commodity. Consequently, the approved quantity of food resources was reduced from 53,840 MT to 46,380 MT. Non-availability of CSB resulted in reduction of food resources in the Safe Motherhood and Child Survival (SMCS) health program (5,670 MT) and Early Childhood Development Center (1,790 MT) education program. In addition, there were gaps in the supply chain due to delays in inland transportation, non availability of wagons etc. As a result there were instances when some partners had no food or had only bulgur or had only oil. Concerning the financial resources, CRS planned for five sources of funding in the original DAP II budget: Development Assistance/Economic Support Fund (DA/ESF) funding ($2.2 million), monetization of approved quantity of food ($6.0 million), sale of empty containers ($0.5 million), recipient contribution ($5.2 million) and CRS’s complementary ($10.6 million) and cost share funding ($2.5 million). However, financial resources under monetizationF 24 and DA/ESF did not materialize. As a result financial resources budgeted for the programs reduced by $8.2 million. CRS filled this gap with its private funds, designating $3.3 million/year in FY2002 and FY2003, $4.6 million in FY2004 and $4.8 million during FY2005. E. Changes in Program Activities and Indicators Food For Work (FFW) activities were proposed in DAP II to undertake school infrastructure improvements. However, this activity was dropped from the beginning of FY2002 as it was decided that it was not advisable to use FFW resources for infrastructural activities in institutional properties. Accordingly, the indicator related to this activity was also dropped. In the DAP II program design, Non-Formal Education (NFE) centers were proposed to provide access to education for children unable to attend formal primary schools due to family responsibilities or for other reasons. Since the aim of the DAP education program is to link all children from NFE into formal school as and when they are ready for the same, it was agreed in FY2003 that the NFE should be treated as an education linking/bridging strategy rather than a parallel system to the existing formal schools. Therefore, the education approach was revised to treat the bridging strategy as part of the education outreach program only. Accordingly, the impact indicator related to NFE was deleted from the DAP monitoring plan. In FY2002, one of the education impact indicators (estimated attendance rate) was dropped as the second impact indicator of survival rate takes into account the retention and progress of the children through the system. During FY2002, baseline levels and the targets set for some 24 Under monetization Title II resources (oil) equivalent to $6.0 million were approved for local sale (monetization). However, the Ministry of Finance did not approve a request to grant an exemption from custom duties on imports for monetization. As a result it was not possible to achieve cost recovery benchmarks specified in USAID regulations. Because of this, the monetization component was dropped in FY2002. At the time of this DAP, the cost recovery benchmark for monetization was still in effect. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 59 of the indicators in education and agriculture sectors were revised using the baseline survey results. Indicator targets were revised a second time during FY2005, given the fact that the FY2004 achievements showed that two indicators in health and one indicator in education had exceeded their LOA targets. F. Baseline and Mid Term CRS carried out baseline surveys in FY02 to set benchmarks and targets for the Agriculture and Education sectors. For the health sector, the baseline levels were estimated using the DAP I final evaluation results and USAID funded NFHS-IIF 25 conducted in 1999. The Mid Term Review (MTR) of DAP II was carried out in FY04. The MTR revealed that the progress was proceeding as per the plan. However, the MTR also highlighted the fact that the linkage between the positive outcomes of the agriculture, health and education programs and household level food security was not sufficiently documented. II. Objectives of the DAP II Final Evaluation The overall objective of the evaluation is to assess and demonstrate the impact of program strategies and interventions on the achievement of the intended results, as measured through indicators developed for each sector (please see Annexure II for indicators). The data collected during the final evaluation will be compared with the baseline estimates and the targets set for each of the indictors. The overall objective and sub objectives of the final evaluation, and key evaluation questions, are given below. Overall objective: To assess and demonstrate the impact of program strategies and interventions in health, education and agriculture sector programs implemented under DAP II during 2002-2006. As part of this objective, the evaluation will also assess how program management and implementation affected program achievements Sub objectives: 1. To carry out a comparative analysis between baseline and final evaluation surveys, assess change in indicators for health, agriculture, and education in program communities and program participants 2. To identify program strategies, structures, systems and interventions that contributed to or impeded the achievement of intended impact of program interventions. 3. To identify external factors beyond the control of the program that affected the relationships between outputs and impacts, such as environmental factors (e.g. rainfall). 4. To assess the effectiveness of CRS and partners’ technical, administrative and managerial systems and approaches to the DAP, and the impact of the same on program outcome indicators. 5. To assess progress made in responding to MTR recommendations 6. To assess the manageability and effectiveness of the Management Information System 7. To assess the sustainability of the programs’ positive impact 8. To develop recommendations for phase out programming in general based on evaluation findings and a review of the Title II Phase Out plan submitted by CRS/India. 25 National Family Health Survey Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 60 9. To develop recommendations about the effective use and disposal of Title II program￾generated income at CP/OP level. 10. To help establish a baseline for key Phase Out targets. Evaluation questions 1. To what extent has the expected change in the indicators (see Annexure II) been achieved compared to the baseline levels? 2. To what extent have external factors affected the achievement of program impact? 3. To what extent have internal factors (interventions, structures, systems) influenced (both positively and negatively) achievement of program impact? 4. To what extent have the capabilities of CRS and its partners affected the achievement of program impact? 5. To what extent have the planned activities been implemented on time? 6. What program activities are sustainable by communities without CRS or partner support and why? What program activities do not appear to be sustainable and why? Are there specific activities that can be phased over to government support? 7. To what extent have the MTR recommendations relevant for DAP II been implemented? 8. What strategies can be adopted in the phase out periodF 26 to ensure achievement of program impact in the communities that are yet to achieve results? What options exist for obtaining government or other support to help achieve program impact in these communities? How feasible is the proposed Phase Out plan? 9. To what extent is the monitoring system operational? How manageable, reliable, valid, and useful is data generated by the system? 10. Can the planned impact be achieved in communities that have been recently phased in or are lagging behind within the phase out time frame? What strategies can be adopted to ensure achievement of results in these communities in the given time frame? Are there possibilities of leveraging government support for these recently phased in or lagging communities? 11. Considering partner capacity and the resource context, what is the likely impact of Title II phase out on partners? What do partners believe will be the effect of Title II phase out on them? How do they plan to mitigate these effects?F 27 III. Approach The evaluation will be conducted by an external team comprising of an overall Evaluation Team Leader, a Deputy Team Leader, and Sector Specialists, with data collection assistance from an external survey agency and junior-level researchers. Sector-specific as well as global lenses will be applied during the evaluation. The external Sector Specialists (Sectoral Team Leaders) will also look into qualitative issues at field level for each sector. Three Junior Level Researchers (one per sector) will assist the Sectoral Team leaders with data collection and analysis. The Sectoral Team Leaders will also analyze the quantitative and qualitative information collected by the external survey agency. The Evaluation Team Leader, with support from the Deputy Team Leader, will guide the evaluation. The Team Leader will also 26 In the next cycle of Title II assisted development programming some of the program communities will be phased over to non Title II resources and some will be phased out (programs support will be withdrawn after achieving exit criteria in the communities). While the next phase of programming is in theory is a 5- year program, the current unfavorable climate for food aid has significant implications for the phase out program funding duration. A Title II Phase Out Plan prepared by USAID/India with input from CRS was submitted to Food for Peace in December 2005. 27 This question will be limited to the partners visited by the sector consultants/sector teams Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation 61 synthesize the outputs from the sectoral consultants’ analysis of all information collected, and lead the process of documenting overarching findings and recommendations. For large-scale quantitative data collection, reputed national level survey agency(s) will be hired. Selected agency(s) will implement the survey under the guidance of the final Evaluation Team Leader and sector specialists, with monitoring support from the Junior Level Researchers. CRS’s Technical Advisor for Monitoring and Evaluation will coordinate the evaluation under the guidance of the ACRs-PQ. CRS Sector Coordinators will help provide orientation to the Sectoral Consultants and serve as resource people to the Sectoral Team Leaders as needed during the evaluation process. Methodology The Sector Team Leaders will be responsible for developing the final evaluation design, methods, and tools for each sector, with overall coordination from the Evaluation Team Leader and Deputy and support from the CRS Sector resource people as required. Given the objectives, the final evaluation will use both quantitative and qualitative methods for collecting the required information. In addition, the teams will review reports and other project documents. The quantitative techniques of data collection will be used to assess change in the indicator levels over the program period. This implies that the final evaluation will use the same survey instruments used in the baseline with minor changes to accommodate additional requirements. Survey instruments used in the baseline will be shared with the selected survey agency. Qualitative techniques such as in depth interviews and group discussions will be used by the Sectoral Teams as well as the external survey agency to identify the factors influencing the program impact, understand the effectiveness of the program strategies, structures, systems and interventions and explain the findings emerging from the quantitative data. IV. Schedule As mentioned earlier the results and recommendations of the final evaluation will be used for the review the phase out program strategies. The program proposal review is due towards the end of May, 2006 hence the report of the final evaluation must be finalized by mid May. Please see Annexure III for the schedule. Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 62 Annexure I Health Program Coverage State Office #CPs States covered Safe Motherhood and Child Survival (SMCS) Benefi￾ciaries Bulgur Oil Total food Ahmadabad 3 Gujarat 10,970 197 120 317 Bhopal 7 Madhya Pradesh 8,500 153 94 247 Bhubaneshwar 6 Orissa 59,050 1,063 654 1,717 Guwahati 10 18,202 328 202 530 Arunachal Pradesh, Assam, Manipur, Meghalaya, Nagaland, Tripura & Mizoram Hyderabad 8 Andhra Pradesh 40,000 720 443 1,163 Jaipur 5 Rajasthan 16,319 294 181 475 Lucknow 6 32,589 587 361 948 Uttarp Pradesh, Uttaranchal Pradesh, Himachal Pradesh, Chandigarh Mumbai 5 6,232 112 69 181 Maharashtra, Dadra & Nagar haveli Raipur 3 Chattisgarh 0 0 0 0 Ranchi 6 11,970 215 133 348 Bihar, Jharkhand, part of West Bengal Kolkota 7 West Bengal 36169 651 401 1,052 Total 66 240,001 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 63 Annexure I Contd… Education Program Coverage Early Childhood Development Center (ECDC) School Feeding (SF) Other Child Feeding (OCF) Total (Education) State Office #CPs States covered Benefi￾ciaries Bulgur Oil Total food Benefi- ciaries Bulgar Oil Total food Benefi- ciaries Bulgar Oil Total food Benefi- ciaries Bulgar Oil Total food Ahmadabad 3 Gujarat 3,277 37 9 46 0 0 0 0 16,005 1,050 97 1,147 19,282 1,087 106 1,193 Bhopal 7 Madhya Pradesh 1,730 19 5 24 6,995 220 35 255 9,705 641 59 700 18,430 880 99 979 Bhubaneshwar 6 Orissa 13,633 150 42 192 8,872 278 44 322 13,685 903 83 986 36,190 1,331 169 1,500 Guwahati 10 200 2 1 3 20,016 631 100 731 Arunachal Pradesh, Assam, 24,044 1,587 147 1,734 44,260 2,220 248 2,468 Manipur, Meghalaya, Nagaland, Tripura & Mizoram Hyderabad 8 Andhra Pradesh 9,000 99 27 126 25,000 788 125 913 20,000 1,452 134 1,586 54,000 2,339 286 2,625 Jaipur 5 Rajasthan 2,100 23 6 29 9,291 293 46 339 2,300 152 14 166 13,691 468 66 534 Lucknow 6 7,224 79 22 101 20,708 652 103 Uttar Pradesh, Uttaranchal 755 8,031 530 49 579 35,963 1,261 174 1,435 Pradesh, Himachal Pradesh, Chandigarh Mumbai 5 1,659 18 5 23 3,424 108 17 125 10,246 677 63 740 15,329 803 85 888 Maharashtra, Dadra & Nagar haveli Raipur 3 Chattisgarh 5,775 64 18 82 7,152 225 36 261 6,749 445 41 486 19,676 734 95 829 Ranchi 6 7,390 81 23 104 20,842 657 104 761 Bihar, Jharkhand, part of 24,744 1,633 151 1,784 52,976 2,371 278 2,649 West Bengal Kolkota 7 West Bengal 5211 57 16 73 12,000 387 60 438 21,048 1,389 128 1,517 38,259 1,833 204 2,037 Total 66 57,200 134,300 158,575 350,075 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 64 Annexure I Contd… Agriculture and Humanitarian Assistance Program Coverage State Office #CPs States covered FFW Humanitarian Assistance (HA) Total- all programs Benefi￾ciaries Bulgar Oil Total food Benefi- ciaries Bulgar Oil Total food Benefi- ciaries Bulgar Oil Total food Ahmadabad 3 Gujarat 30,862 2,058 119 2,177 3,707 200 24 224 64,821 3,548 369 3,917 Bhopal 7 Madhya Pradesh 32,999 2,200 126 2,326 2,139 116 14 130 62,068 3,349 333 3,682 Bhubaneshwar 6 Orissa 11,818 788 45 833 9,000 486 60 546 116,058 3,668 928 4,596 Guwahati 10 0 0 0 0 14,394 777 96 873 76,856 3,325 546 3,871 Arunachal Pradesh, Assam, Manipur, Meghalaya, Nagaland, Tripura & Mizoram Hyderabad 8 Andhra Pradesh 0 0 0 0 8,000 432 53 485 104,000 3,491 782 4,273 Jaipur 5 Rajasthan 94,498 6,300 362 6,662 300 16 2 18 124,808 7,078 611 7,689 Lucknow 6 8,815 588 34 622 7,213 390 48 438 84,580 2,826 617 3,443 Uttar Pradesh, Uttaranchal Pradesh, Himachal Pradesh, Chandigarh Mumbai 5 7,800 520 30 550 3,996 216 27 243 33,375 1,651 211 1,862 Maharashtra, Dadra & Nagar haveli Raipur 3 Chattisgarh 43,036 2,869 165 3,034 950 51 6 57 63,662 3,654 266 3,920 Ranchi 6 35,315 2,354 135 2,489 Bihar, Jharkhand, part of West 11,028 596 73 669 111,289 5,536 619 6,155 Bengal Kolkota 7 West Bengal 5,577 372 21 393 46274 2,499 308 2,807 126,280 5,346 934 6,280 Total 66 270,721 107,001 967,798 49,688 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 65 Annexure II 1. Health Sector Indicators Impact: Percentage of children under 2 years whose weight for age falls below 2SD Monitoring: Number of VHW, TBA and RMPs trained in health and nutrition Percentage of children 6-23 months, pregnant women and nursing mothers (up to 6 months lactation) in the program villages, enrolled in the program Percentage of program villages that conduct monthly health and nutrition education sessions every month Percentage of children under 3 years of age who are registered in the program growth monitored every month Percentage of children (12-23 months) who are fully immunised by age one Percentage of mothers delivered in the past year who received 90+ IFA tablets Percentage of program villages without CARE ICDS overlap or with agreements Percentage of program villages where health data collection is conducted in catchment areas and developed exit standards Percentage of children receiving semi solid food between 6-9 months 2. Education Indicators Impact: 1. Survival rate disaggregated by gender a. ECDC to Primary b. Primary: class I to class IV 2. Net enrolment rate in CRS outreach program catchment areas by gender Monitoring: Number of children receiving Title II school meal Number of education outreach programs implemented Percentage of trained education providers using child centered methods in classrooms 3. Agriculture Indicators Impact: Percentage of cultivable area producing more than one crop in a year Number of weeks that water is retained in pre selected wells in a year Monitoring: Average area irrigated (ha) per project disaggregated by project duration Percentage of watershed projects with watershed committees having operational guidelines for water distribution and systems to monitor Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 66 Annexure III Time Schedule of Evaluation activities Sl.N o. Activity Output Who How By when 1 Draft Scope of work Draft ready CRS Internal Jan 11 2 Review scope of work Revised version ready to share CRS/ USAID-India e mail Jan 18 3 Finalize scope of work Final version of Scope of work ready CRS Internal Jan 25 4 Shortlist consultants Identify consultants & discuss CRS Internal Jan 16 Hiring of consultants Contract signed CRS/Sectoral Consultants/Evaluation Team Leader & Deputy Internal/Mail January 25 5 Orientation to consultants, finalize study design including sampling design Relevant material shared with consultants, orientation workshop conducted CRS/Sectoral consultants/Team Leader and Deputy/USAID-India e mail, Workshop (Feb 2-3) Jan 16- Feb 3 6 Draft scope of work for survey agency Survey SOW CRS Internal Feb 3 7 Sample selection List of program communities/institutions ready CRS Internal Feb 10 8 Share Survey SOW with agencies SOW sent to agencies CRS e mail Feb 4 9 Bids from survey agencies Bids received & reviewed by bid committee CRS/Survey agencies mail Feb 9 10 Short listing of survey agency Bid committee recommends an agency (2nd meeting) CRS Bid Committee Internal Feb 10 11 Draft tools for data collection & field testing Agency submits drafts to CRS Survey agency Internal, e mail, field visit Feb 17 12 Review draft tools submitted by survey agency Feedback given to agency Survey agency/CRS/Sectoral consultants Internal, e mail Feb 21 13 Finalize tools Final tools submitted to CRS Survey agency e mail Feb 22 14 Field survey Agency completes survey activities in all selected areas Survey agency Field visits Feb 24- March 25 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 67 15 Sector team field visits Sector teams complete field visits Sector teams/CRS Field visit February 12- to March 31 16 Survey monitoring Survey teams visited by CRS/ consultants (Jr Researchers) CRS/Consultants (Jr Researchers) Field visits Feb 24- March 25 17 Draft analysis plan Agency submits draft Survey agency Internal, e mail March 1 18 Review draft analysis plan submitted by survey agency Feedback given to survey agency CRS/Sectoral Consultants Internal/e mail March 7 19 Finalize analysis plan Agency submits Revised analysis plan to CRS Survey agency/CRS/Sectoral Consultants Internal, e mail March 9 20 Data entry and analysis (draft tables based on partial data) Agency submits draft tables Survey agency Internal, e mail March 15 21 Review draft tables Feedback from CRS/Sectoral Consultants given to agency Survey agency/CRS/Sectoral consultants Internal, e mail March 22 22 Submit final tables Agency submits tables based on complete data to CRS Survey agency e mail March 31 23 Draft sector reports Sectoral Consultants submit sector reports to CRS Sectoral Consultants e mail April 10 24 Review draft report Feedback from CRS/Evaluation Team Leader and Deputy given to Sectoral Consultants (separate, consolidated feedback to be given by CRS and by Team Leader) CRS/Evaluation Team Leader and Deputy e mail April 15 25 Draft Combined report – first draft Evaluation Team leader submits Agency Report – first draft Evaluation Team Leader with Deputy e mail April 21 26 Revise sector reports Sectoral Consultants submit revised sector reports - 2nd draft Sector Consultants e mail April 26 27 Review Combined Report Feedback from CRS given to Evaluation Team Leader and Deputy CRS e mail/2nd visit/1 day workshop May 1 28 Revise Combined report Evaluation Team Leader submits revised Agency Report – 2nd draft Evaluation Team Leader with Deputy e mail May 5 29 Review revised draft reports Feedback from CRS/USAID given to Sectoral Consultants / Evaluation Team Leader CRS/Evaluation Team Leader / USAID-India e mail May 12 30 Final revision of reports (Combined & sector reports) Sectoral Consultants & Team Leader submit final Reports Sectoral Consultants/Evaluation Team Leader with Deputy e mail May 20 Note: Consultants=sector consultants Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 68 Indicator Tracking Table for the Health Sector Base Line FY02 FY02 FY02 FY03 FY03 FY03 FY04 FY04 FY04 FY05 FY05 FY05 Final Evaluation Indicator Indicator Description FY02 Target Achd. % Achd Target Achd. % Achd Target Achd. % Achd Target Achd. % Achd Target Achd. % Achd SO1: Improve health of 240,000 pregnant/lactating women and children aged 0-3 years Impact 1. % children under 2 years whose weight-for-age falls below -2SD 47 - - - - - - - - - - - - 36 37 98 Monitorin g 1. Number of VHW, TBA and RMPs trained in health & nutrition 1200 3600 2245 62% 3600 4757 132% 3600 4753 132 3600 5593 155 14,400 17,348 120 2a. % children 6-23 m, pregnant women and nursing mothers (up to 6 months lactation) in the program villages, enrolled in the program 20 - - - - - - 30 83 276 - - - 83 68.1 82 2b. % program villages that conduct monthly health and nutrition education sessions every month 44 50 91 182% 60 78 130% 70 78 111 80 81 101 90 81 90 3. % children under 3 years of age who are registered in the program growth monitored every month 78 80 83 104% 90 85 94% 90 77 86 90 83 92 90 83 92 (outcome) 4a. % children (12-23m) who are fully immunized by age one 40 - - - - - - 45 28 62 - - - 50 53 106 (outcome) 4b. % mothers delivered in the past year who received 90+ IFA tablets 24 - - - - - - 29 63 218 - - - 63 36 57 5a. % program villages without CARE ICDS overlap or with agreements - - - - - - - 100 100 100 100 100 100 100 100 100 5b. % program villages where health data collection is conducted in catchments area and developed exit standards 0 25 16 64% 50 54 108% 100 98 98 100 98 98 100 98 98 6. %children receiving semi solid or solid food between 6-9 months 70 - - - - - - - - - - - - 75 77 103 Note: the FY05 targets and achievements of the annual reporting indicators have been used as the targets and achievement for the final evaluation. Note:Baseline- Coverage for SMCS services increased from partial coverage under DAP I to complete coverage of revenue village in DAP II. Therefore, the baseline figures were estimated using DAP I final evaluation, USAID funded NFHS II survey of 1999 and the state wise CRS resources level excepting for the indicator number 6. Baselines for the monitoring indicators numbered 1,2b&3 are estimates based on the FY2001 achievements and the number of villages that will phased out in 2001 and number of new villages to be phased in during FY2002 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 69 Indicator Tracking Table for the Agriculture sector Note: the FY05 targets and achievements of the annual reporting indicators have been used as the targets and achievement for the final evaluation. 28 If same land is irrigated from one irrigation structure in two different crop seasons the land area will be counted twice Base line FY02 FY02 FY02 FY03 FY03 FY03 FY04 FY04 FY04 FY05 FY05 FY05 Final Evaluation Indicator Type Indicator Description FY02 Target Achvd % achvd vs target Target Achvd % achvd vs target Target Achvd % achvd vs target Target Achvd % achvd vs target Target Achvd % achvd vs target SO2: Increased agricultural productivity of 40,000 farming households in 200 communities during the DAP Impact 1. % Cultivable area producing more than one crop in a year 9 - - - - - - - - - - - - 31 32 103 2. Number of weeks that water is retained in pre-selected wells in a year 31 - - - - - - - - - - - - 41 31 76 1. Average area irrigated (ha) per project desegregated by project durationF28 20 35 20 57% 35 7 20% 50 14 28 65 17 26 80 17 21 Monitoring 2. % watershed projects with watershed committees having operational guide-lines for water distribution and systems to monitor 0 15 17 113% 40 18 45% 60 29 48 100 43 43 100 43 43 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 70 Indicator Tracking Table for the Education Sector Base line FY02 FY02 FY02 FY03 FY03 FY03 FY04 FY04 FY04 FY05 FY05 FY05 Final Evaluation Type Indicator Description FY02 Target Achvd % achvd vs target Target Achvd % achvd vs target Target Achvd % achvd vs target Target Achvd % achvd vs target Target Achvd % achvd vs target SO3: Increase opportunities for 350,075 disadvantaged children (SC/ST/OBC) annually, especially girls, to participate in quality primary education Impact 1. Survival rate by grade disaggregated by gender ECDC to primary Total 52 - - - - - - 55 55 100 - - - 60 64 107 Girls 53 - - - - - - 55 53 96 - - - 60 67 112 Primary (I-IV)F29 Total 65 - - - - - - 78 84 108 - - - 70 88 126 Girls 67 - - - - - - 78 84 108 - - - 70 87 124 2. Net enrollment rate in CRS outreach program catchment areas by gender Total 58 - - - - - - - - - - - - 85 90 106 Girls 33 - - - - - - - - - - - - 85 91 107 Monitori ng 1.Number of children receiving title II school meal 298,475 350,075 275,387 79 350,075 288403 82 350,075 336,978 96 350,075 306,281 88 350,075 306,281 88 2.Number of education outreach programs implemented 17 25 22 88 25 44 176 50 97 194 97 106 109 97 106 109 3.% trained education providers using child centered methods in classrooms 25 50 51 102 55 48 87 60 57 95 65 64 99 70 64 91 Note: the FY05 targets and achievements of the annual reporting indicators have been used as the targets and achievement for the final evaluation. 29 In case of Mid term the survival rate will assess the percentage of children enrolled in class 3 among those enrolled in class 1 in FY02 and Mid term target for the survival rate will be set based on the baseline estimate for the survival rate up to class 3 and the target for the survival rate up to class 4 Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 71 Annex II: CRS/India Organigram Country Representative Marc D’Silva Deputy Country Rep Alex Mathew Deputy Country Rep. Clodagh McCumiskey Asst. Country Rep. Management Quality Tsegaya Kassa Department Director Internal Audit Vacant Asst. Country Rep. Program Quality Rekha Abel State Representative Hyderabad Sanjay Singh State Representative Chennai Fr. D.M Celestine State Representative Raipur Arun Pandey State Representative Bhopal Kishore Patnaik State Representative Ahmedabad K. S. Venkatesh State Representative Mumbai Shivakamy Iyer Asst. Country Rep. Program Quality Sarah Cashore State Representative Lucknow Kushal Neogy State Representative Jaipur Suvabrata Dey State Representative Ranchi David Solomon State Representative Guwahati Gopen Moses State Representative West Bengal Sanchita Banerjee State Representative Bhubaneswar Kasturi Sengupta Department Director Finance Hitesh Somani Department Director Material Resources Dharmesh Kumar Department Director Admin & HR Vacant Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 72 Details of the Evaluation Methodology: Evaluation Methodology The Final Evaluation was led by a team of four external experts. Three Junior Researchers and two reputed national level survey agencies were hired to assist the evaluation team. CRS’s Technical Advisor for Monitoring and Evaluation assisted the external team in coordination and logistics of the evaluation activities under the guidance of the Assistant Country Representatives for Program Quality. The evaluation used both quantitative and qualitative methods for collecting data. External experts also visited program sites and interacted with the partners and communities to study the program organization, implementation and management. Quantitative surveys were used to assess change in the indicator levels over the program period. Survey instruments used in the baseline survey were repeated in the final evaluation to facilitate comparative analyses. Qualitative techniques such as key informant interviews, group discussions and activity observations were used to identify the factors influencing program impact and to explain findings emerging from the quantitative data. Design and sample size CRS/India’s Title II assisted DAP II reached 976, 830 program participants in FY ‘05. The evaluation covered 10 out of 11 CRS State Office programs. These State Offices accounted for 87% of the program’s resources. The states covered in the evaluation include Andhra Pradesh, Bihar, Chhattisgarh, Gujarat, Jharkhand, Madhya Pradesh, Maharashtra, Orissa, Rajasthan, Uttar Pradesh and West Bengal. Meghalaya and Assam were also included in a sector consultant visit. The sample sizes for the individual sectors were worked out to measure an all India level change of five percent in the indicators with 95 percent confidence and 80 percent power. However, in some cases where the expected change was different (other than 5 percent), the sample sizes were calculated accordingly. The design effect of 1.3 to 2 percent has been taken into consideration in the process of establishing the sample sizes. The sample sizes were adjusted to accommodate non-response. The extent of non-response was estimated to vary between four to eight percent depending on the target group (respondent type). The program villages and the respondents were selected by a stratified systematic probability, proportional to size, using the same multi-stage cluster sampling methodology followed in the baseline survey. In selected health program villages, listing of 0-23 months children was carried out to select the mothers. In agriculture, farmers were stratified according to their land location: medium upland, medium low land, and low land, and selected based on land size. In Education, the institution based sample was selected using the attendance registers and for household sample, listing of 6-14 years children was carried in the selected outreach communities. A total of 7,078 children from 506 institutions and 44 villages spread across 10 states and over 30 districts were covered in education sector evaluation survey. With regard to the number of partners, the evaluation sample was spread across 21 CPs. Participants include 4,532 children under ECDC, SF and OCF. Out Reach Programs were studied in 44 villages (Out Reach communities). The study has been conducted at two levels- 777 families with children in 6-14 age group and 1,475 children from the schools located in these 44 villages on Catholic Relief Services/India Title II Development Assistance Program II (2002-2006) Final Evaluation Page 73 a sample basis (10 children per school). In addition, 294 children who were attending less than 80 school-days participated in the education portion of the end line survey. A total of 3,546 mothers of children 0-23 months in health and 1,080 farmers in agriculture were covered by the sample from 344 villages and 506 institutions spread across 11 states and over 80 districts. With regard to the number of partners, the evaluation sample was spread across 30 CPs in health and 20 CPs in agriculture. Survey implementation and data processing Two national survey agencies were hired to conduct the final evaluation survey. The Agriculture Finance Corporation Ltd carried out the agriculture survey and TNS India Pvt. Ltd carried out survey of the Education and Health programs. A total of 126 interviewers and supervisors conducted the survey. The number of interviewers per team and the gender composition varied depending on the respondent type. For instance, the health sector survey teams had three female interviewers and a male supervisor in each team and for education survey the survey teams were comprised of two male interviewers. The survey teams were given three to five days of classroom training and field practice. Senior agency staff monitored the survey and Junior Researchers monitored the fieldwork. The completed questionnaires were then sent for data processing in Delhi where trained staff carried out editing, data entry, data cleaning and tabulation. Software packages of Foxpro, Quantom and SPSS were used for data processing. More details about the field visits completed by the sector consultants including specific locations visited and groups or individuals interviewed can be found in the sector reports annexed to this document. Presentation of Results In this report baseline data and data from the final evaluation (FE) surveys are compared, where both baseline and final evaluation data is available. As required, CRS has baseline data for all monitoring and impact indicators, however some interesting information has emerged from the final evaluation that does not always have accompanying baseline data. Furthermore, all data presented in this report is condensed to make the document more reader-friendly. Source data tables can be found in the accompanying sector reports in the annex. Finally, for the health program data has been measured using several different parameters one of which is length of time the community in which the respondent lives has been participating in the SMCS program (one, two or three years). Although in many cases respondents from villages that participated for three years had far superior results, for the purposes of the report, the result used to represent the final evaluation is the total response for all villages regardless of how many years the community participated in the program. Therefore the data reflects the responses of mothers who may have recently enrolled in the program or who may already have been in the program for three years.