ASSESSMENT OF THE GLOBAL HEALTH OUTCOME MONITORING DEMONSTRATION IN TWO COUNTRIES: MADAGASCAR AND GUATEMALA December 2007 This publication was produced for review by the United States Agency for International Development. It was prepared by William Vargas, Jennifer Luna, and Pierre-Marie Metangmo through the Global Health Technical Assistance Project. ASSESSMENT OF THE GH OUTCOME MONITORING DEMONSTRATION IN TWO COUNTRIES: MADAGASCAR AND GUATEMALA DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 07-001-51) is available online. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources/. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. Assessment of GH Outcome Monitoring Pilot v CONTENTS ACRONYMS ................................................................................................................................................ vii EXECUTIVE SUMMARY .............................................................................................................................. 1 I. INTRODUCTION ........................................................................................................................................ 5 BACKGROUND ........................................................................................................................................ 5 PURPOSE OF THE ASSESSMENT ........................................................................................................ 6 OVERVIEW OF USAID MATERNAL AND CHILD HEALTH AND REPRODUCTIVE HEALTH PROGRAMMING IN GUATEMALA AND MADAGASCAR ...................................................................... 6 METHODS AND DATA SOURCES ........................................................................................................10 II. FINDINGS OF THE ASSESSMENT .......................................................................................................11 FEASIBILITY OF CONDUCTING LQAS-BASED OUTCOME MONITORING (OM) ANNUALLY .........11 Can OM be carried out in country? ................................................................................................ 11 Is OM timely? ................................................................................................................................. 11 Is OM cost-effective? ..................................................................................................................... 14 Is OM precise? ............................................................................................................................... 15 USEFULNESS OF THE OM DATA FOR PROGRAM MANAGEMENT, DECISIONMAKING, AND REPORTING BY MISSIONS AND IN-COUNTRY PARTNERS .............................................................17 Purpose of OM: .............................................................................................................................. 17 How OM can be used: .................................................................................................................... 18 Limitations of use of OM approach: ............................................................................................... 19 ASSESSING LOCAL TECHNICAL CAPACITY IN USING LQAS FOR OM ..........................................19 STRATEGIES FOR TRANSFERRING THE OM APPROACH TO MISSIONS AND IN-COUNTRY PARTNERS AND FOR EXPANSION TO OTHER MISSIONS ..............................................................21 III. RECOMMENDATIONS ..........................................................................................................................23 Classic LQAS ................................................................................................................................. 25 30x10 Cluster Methodology ........................................................................................................... 26 IV. APPENDICES ........................................................................................................................................27 APPENDIX A: SCOPE OF WORK ..............................................................................................................27 APPENDIX B: INTERVIEW GUIDES FOR GUATEMALA ..........................................................................33 APPENDIX C: PERSONS AND ORGANIZATIONS CONTACTED ............................................................41 APPENDIX D: REFERENCE MATERIALS/DOCUMENTS CONSULTED .................................................45 APPENDIX E: LIST OF OM INDICATORS ................................................................................................. 47 APPENDIX F: GLOBAL HEALTH OUTCOME MONITORING (OM) SURVEYS ....................................... 51 APPENDIX G: GUATEMALA AND MADAGASCAR–KEY INDICATORS COMPARISON OF CI ............. 55 APPENDIX H: REFERENCES ....................................................................................................................59 vi Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot vii ACRONYMS ADRA Adventist Development and Relief Agency International AED Academy for Educational Development AIDS Acquired immune deficiency syndrome ANC Antenatal care CI Confidence interval CRS Catholic Relief Services CSHGP Child Survival and Health Grants Program CSTS+ Child Survival Technical Support Plus CTO Cognizant technical officer DPT Diphtheria, pertussis, tetanus F Director of Foreign Assistance FACTS Foreign Assistance Coordination and Tracking System FAF Foreign Assistance Framework FANTA Food and Nutrition Technical Assistance FP Family planning FSN Foreign service national GH Global health HC Health center HIP Hygiene Improvement Project HIS Health information system HIV Human immunodeficiency virus HKI Helen Keller International HPN Health, population, nutrition IEC Information, education, and communication IMCI Integrated Management of Childhood Illness IPT Intermittent preventive treatment ITN Insecticide-treated net IYCF Infant and young child feeding KPC Knowledge, practice, and coverage LLTN Long-lasting treated net LOE Level of effort LQAS Lot Quality Assurance Sampling M&E Monitoring and evaluation MCDI Medical Care Development International MCH Maternal and child health MCHC Maternal and child health care MCHN Maternal and child health and nutrition MICS Multiple Indicator Cluster Survey MOH Ministry of Health MSPAS Ministry of Public Health and Social Assistance NGO Nongovernmental organization OM Outcome monitoring OP Operational Plan ORS Oral rehydration solution viii Assessment of GH Outcome Monitoring Pilot ORT Oral rehydration therapy PDA Personal digital assistant PMA Program Management Area PNUD/GUA Programa de Naciones Unidas para el Desarrollo, Guatemala PP Postpartum PSI Population Services International PVO Private voluntary organization RFP Request for Proposal RH Reproductive health SA Supervision area SD Standard deviation SOW Scope of work SP Sulfadoxine-pyrimethamine STI Sexually transmitted infection TA Technical assistance TOST Training of survey trainers TT Tetanus toxoid UNDP United Nations Development Program USAID United States Agency for International Development USG United States Government WAZ Weight for Age Z-score WHO World Health Organization Assessment of GH Outcome Monitoring Pilot 1 EXECUTIVE SUMMARY The Bureau for Global Health (GH) of the United States Agency for International Development (USAID) commissioned this assessment to examine two demonstration experiences of outcome monitoring (OM) implemented by the Food and Nutrition Technical Assistance (FANTA) Project in Guatemala and Madagascar. The overall aim of this work was to provide recommendations to USAID about the feasibility and usefulness of outcome monitoring for USAID programs. The demonstration studies focused on measuring population-level outcome indicators for maternal and child health and family planning. Fieldwork for both Guatemala and Madagascar took place from June to August 2007. A three-person team was assembled for this assessment, which took place from September to October 2007. The purpose of this assessment was to: 1. Assess the feasibility of conducting lot quality assurance sampling (LQAS)-based OM annually 2. Assess the usefulness of the data for program management, decision-making, and reporting by USAID Missions and in-country partners 3. Assess local capacity issues related to the transfer of the approach to Missions and in-country partners over the medium term 4. Make recommendations to roll out the OM process to USAID Missions In order to obtain information for this assessment, the team interviewed key individuals and groups and reviewed documents. Interviews were conducted with USAID/GH staff; USAID Mission staff in Guatemala and Madagascar; FANTA staff in Washington, D.C.; in-country survey implementers; Mission partner organizations; and Ministry of Health personnel. The findings included in this report represent the opinions of the people interviewed as well as the recommendations of the assessment team. The main finding from this assessment is that all who were interviewed value OM and want it repeated; in addition, they all have recommendations that should be put into place to streamline the process. Partners in particular would like a participatory process that involves them at all stages from data collection to data use. Key findings that correspond to the purpose of the assessment are briefly described below. Feasibility It is feasible to implement OM surveys in country, but the process should be streamlined in the future. Surveys in both Guatemala and Madagascar were carried out successfully. However, the current process takes six to seven months from preparation of fieldwork to production of the final report. This can be shortened by improving the planning and implementing process; adjusting questionnaires so they contain fewer variables; making sure that final versions of questionnaires are ready before interviewer training begins; preparing training material in 2 Assessment of GH Outcome Monitoring Pilot advance of training sessions; improving personal digital assistant programs; and streamlining the analysis process. The cost of these studies would be lower with a streamlined process. For these demonstrations, FANTA used a modified approach to LQAS. The modification introduces an element of clustering to reduce costs of data collection in the field. In each supervision area, seven clusters were chosen, using probability proportional to size sampling, and three interviews were conducted in each cluster for a total of 21 interviews per supervision area. Computer simulations were performed on this modification which indicated its validity based on low inter- and intra-cluster correlations. This assumption will be tested with actual data from Guatemala and Madagascar. After this testing is complete, a decision to promote OM with the modified approach or to use conventional LQAS should be made. Usefulness Missions and implementing partners find OM useful and want it repeated. Missions in both Guatemala and Madagascar will use the information for management decisions and for reporting. They will make funding allocation decisions based on this information. OM information is most useful for management decisions if it has a geographic focus, which means that LQAS lots correspond to discrete geographic areas. Partners in Guatemala proposed an important way the country can benefit from OM information. They proposed that partner organizations come together to discuss OM findings and then collectively decide how each organization can contribute to improving these indicators. OM as implemented in Madagascar and Guatemala cannot be used to judge the performance of individual implementers. For this, it is better to use each organization’s own monitoring and evaluation system. In addition the outcome measurements could be used to calculate the number of lives saved based on the methodology that is used by the Child Survival and Health Grants Program (CSHGP) that is based on the methodology used in the Lancet series on child health. This methodology can be accessed on the Child Survival Technical Support Project Plus website1 Local Capacity OM can be implemented by local partner organizations. Both Guatemala and Madagascar have in-country data collection capability, as evidenced by the work of FANTA subcontractors in each country who did the fieldwork. Analysis could also be performed in country with streamlined processes, training, and limited technical assistance support from a central level. Transferring the OM Approach to Mission and In-Country Partners and Expansion to Other Missions Transferring the OM approach in Guatemala and Madagascar and expanding it to other Missions can be accomplished by developing tools and guidance based on the recommendations from 1 http://www.childsurvival.com/start.cfm Assessment of GH Outcome Monitoring Pilot 3 these demonstration efforts and by encouraging a participatory process that focuses on monitoring and not external oversight. The following actions would help transfer OM capacity to Missions and in-country partners and expand OM to other Missions: Develop tools and guidance based on the recommendations from the demonstration efforts and considering previous experiences of organizations that have used LQAS and Cluster sampling methodologies. Develop guidance emphasizing that the focus of OM is on monitoring and not exclusively on external oversight, and that therefore it should be a participatory process. Maintain limited technical support at a central level for questions regarding implementation of OM surveys and to ensure that Missions understand adjustments made to the survey due to changes in technical areas. OM is a rapid population-based assessment. It is important to clarify that this process is for monitoring and not just for external oversight. Concentrating on the monitoring aspect permits a participatory process at all stages, including design, data collection, data analysis, and decision￾making. Members of local implementing partners could participate in the data collection if properly supervised so that they follow a standard sampling protocol. This would increase buy-in of implementing partners, increase the credibility of findings with partners, and improve partners’ use of this information for decision-making. In addition, sustainability of the effort would be possible, since these organizations would be able to carry out the process in the future. 4 Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot 5 I. INTRODUCTION BACKGROUND This assessment was commissioned by the Bureau for Global Health (GH) of the United States Agency for International Development (USAID) to examine two demonstration experiences of outcome monitoring (OM) implemented by the Food and Nutrition Technical Assistance (FANTA) Project in Guatemala and Madagascar. These demonstration studies focused on measuring population-level outcome indicators for maternal and child health (MCH) and reproductive health (RH), including family planning (FP). For in-country implementation of the surveys, FANTA subcontracted two local groups: CIENSA in Guatemala and PENSER in Madagascar. Fieldwork for both Guatemala and Madagascar took place from June to August 2007, with each study’s final report expected at the end of November 2007. A three-person team was assembled for this assessment, which took place from September to October 2007. GH became interested in exploring annual OM for the following reasons: To provide USAID Missions with outcome information that is specific to geographic areas where USAID’s activities are implemented between Demographic and Health Surveys (DHS), which occur approximately every three to five years To provide complementary data to the output monitoring required as part of the foreign assistance reform USAID has traditionally measured progress in the health sector through internationally recognized outcome and impact indicators such as contraceptive prevalence rates, immunization rates, proportion of births with skilled attendance, and use of bednets. This information is usually obtained through DHS and other large surveys, which are expensive and are conducted only about once every five years. However, this information needs to be collected on a more regular basis (annually if possible), together with annual output indicators to provide good information for Missions to make programmatic decisions and for reporting progress to Washington. In September 2006, GH sponsored a one-day expert consultation to explore the feasibility and cost-effectiveness of collecting outcome data on an annual basis. GH decided to demonstrate the use of lot quality assurance sampling (LQAS) methodology for annual OM principally because it can generate both programwide summary statistics and lot-specific information potentially useful for program management and because its cost appeared reasonable in comparison with other methods. In December 2006, GH initiated a discussion with the FANTA Project, managed by the Academy for Educational Development (AED), regarding the collection of FP/RH, MCH, and malaria outcome data using a modified LQAS approach on a trial basis in Guatemala and Madagascar. This resulted in the two demonstration studies. FANTA was chosen because it was already implementing a similar survey in Madagascar. For their work in Madagascar FANTA had developed a modified LQAS methodology. The modification introduces an element of clustering to reduce costs of data collection in the field. In each supervision area, seven clusters were chosen, using probability proportional to size 6 Assessment of GH Outcome Monitoring Pilot sampling, and three interviews were conducted in each cluster for a total of 21 interviews per supervision area. The survey included indicators for five different age groups: 1. Mothers of children 0–6 months 2. Mothers of children 0–11 months 3. Mothers of children 12–23 months 4. Mothers of children 0–59 months 5. Women of reproductive age (15–49 years) In order to accommodate these different age groups, five different questionnaires were developed. Each questionnaire was applied 3 times in each cluster. There were 7 clusters in each supervision area; therefore a total of 21 questionnaires were applied in each supervision area. In the case of Guatemala, there were five supervision areas. There were a total of 105 interviews for each age group. The total number of interviews for the entire survey area was 525. For Madagascar, there were four supervision areas for a total of 84 interviews per age group. The total number of interviews was 420 for the entire survey. PURPOSE OF THE ASSESSMENT The purpose of this assessment was to: Assess the feasibility of conducting LQAS-based OM annually Assess the usefulness of the data for program management, decisionmaking, and reporting by USAID Missions and in-country partners Assess local capacity issues related to the transfer of the approach to Missions and in￾country partners over the medium term Make recommendations to roll out the OM process in USAID Missions This assessment serves as input to decisions about 1) whether LQAS is a feasible and appropriate methodology to use for annual or biannual monitoring of outcome indicators; 2) how to maximize the usefulness of the results for Mission programming; and 3) how to transfer capacity to implement the OM approach to Missions and in-country partners (see appendix A). The consultant team was asked to make recommendations that went beyond just assessing FANTA’s modified LQAS technique. Almost all of the findings should be taken into consideration even if the decision is to use classic LQAS (instead of modified LQAS) or cluster sampling methodology. OVERVIEW OF USAID MATERNAL AND CHILD HEALTH AND REPRODUCTIVE HEALTH PROGRAMMING IN GUATEMALA AND MADAGASCAR The following brief descriptions of programs in Guatemala and Madagascar provide background for understanding the assessment of the demonstration studies. Assessment of GH Outcome Monitoring Pilot 7 Guatemala: USAID concentrates MCH and RH activities in the Western Highlands (see map). The following table briefly describes USAID/Guatemala’s health programs: Implementing Partner Description APROFAM Network of FP and MCH clinics in most of country; affiliate of International Planned Parenthood Federation; will graduate from USAID funding September 30, 2009. United Nations Development Program (UNDP) Public International Organization grant to provide technical assistance (TA) to Ministry of Public Health and Social Assistance (MSPAS) and to manage grants to nongovernmental organizations (NGOs) contracted by MSPAS to work on the extension of services program Calidad en Salud (University Research Co.-Quality Assurance) Contract to provide TA to MSPAS Asociación SHARE Title II with some MCH Catholic Relief Services (CRS) Title II with MCH Save the Children Title II with MCH Alianza (RTI International￾Alliance) Arranges alliances with private funds for health and education activities MSPAS Partner through TA from Calidad en Salud and UNDP 12 convenios 11 convenios 2 convenios 2 convenios 1 convenio 6 convenios 2 convenios 8 Assessment of GH Outcome Monitoring Pilot Madagascar: The following table briefly describes USAID programming in Madagascar. Implementing Partner Description SanteNet SanteNet, USAID/Antananarivo’s Strategic Objective 5 bilateral program, provides services and products through community￾based agents linked to the primary health centers in four of Madagascar’s six provinces. Population Services International (PSI) PSI uses social marketing campaigns to promote the availability and use of health￾based products (contraceptives, malaria treatment and bednets, sexually transmitted infection (STI) treatments, water purifiers). Medical Care Development International (MCDI) Child survival grants to support focused interventions in 53 communes Voahary Salama Consortium of 40 local NGOS with health activities Hygiene Improvement Project (HIP) Implements projects of water and sanitation Adventist Development & Relief Agency International (ADRA) Title II with child survival CRS Title II CARE Title II Assessment of GH Outcome Monitoring Pilot 9 10 Assessment of GH Outcome Monitoring Pilot METHODS AND DATA SOURCES In order to obtain information for this assessment, the team interviewed key individuals and groups and reviewed documents. Interviews were conducted with USAID/GH staff; USAID Mission staff in Guatemala and Madagascar; FANTA staff in Washington, D.C., who were involved in the demonstration surveys; survey implementers and Mission partner organizations in Guatemala and Madagascar; and Ministry of Health (MOH) personnel in each country. Interview guides developed by the assessment team were used for this work (see appendix B). William Vargas and Pierre-Marie Metangmo traveled to Madagascar from October 2 to October 10, and Vargas and Jennifer Luna traveled to Guatemala from October 14 to October 20. The assessment team gave two presentations to USAID. The first was a preliminary presentation (October 12, 2007) on the findings from the Madagascar trip. The second presentation on the overall findings from both Guatemala and Madagascar was held at USAID on October 24, 2007. The resulting findings represent the opinions of the people interviewed about the demonstration experiences as well as the opinions and recommendations of the assessment team. Interviews: In Guatemala, staff from the following organizations were interviewed (see appendix C for details of individuals): USAID/Guatemala CIENSA: Implemented pilot OM survey through a subcontract with FANTA Implementing partners of USAID/Guatemala: APROFAM; UNDP; Calidad en Salud; MSPAS; Title II (Asociación SHARE, CRS, and Save the Children) In Madagascar, staff from the following organizations were interviewed (see appendix C for details of individuals): USAID/Madagascar PENSER: Implemented pilot OM survey through a subcontract with FANTA Implementing partners of USAID/Madagascar: SanteNet; PSI; MCDI; Voahary Salama; HIP; Title II (ADRA, CRS, and CARE) Documents: The assessment team reviewed a variety of documents from the GH Tech and FANTA projects (see appendix D), including presentations from the 2006 expert consultation meeting; project descriptions from the Guatemala and Madagascar USAID Missions; subagreements with CIENSA and PENSER; preliminary results from pilot studies for both countries; description and computer simulations of modified LQAS; lists of GH indicators;2 report from Layers I Madagascar (survey previously implemented by FANTA in Madagascar for Title II programs); USAID’s description of the OM pilots;3 questionnaires from Layers; and training material developed by CIENSA. A presentation on the OM pilots was made by FANTA staff to the team and USAID staff at the beginning of the assessment. 2 See appendix E. 3 See appendix F. Assessment of GH Outcome Monitoring Pilot 11 II. FINDINGS OF THE ASSESSMENT Both the Guatemala and Madagascar USAID Missions see the information from OM as being very useful for program management and reporting. The implementing partners want OM surveys repeated, but they have suggestions for improving the process and better involvement of partners. The assessment team also recommends repeating the surveys, but with suggestions to streamline the process, reduce costs, and increase local ownership and use of information. The following describes the specific findings of the assessment in terms of feasibility of annual OM, usefulness of OM for program management, and strategies for transferring OM to other Missions and in-country partners. FEASIBILITY OF CONDUCTING LQAS-BASED OUTCOME MONITORING (OM) ANNUALLY In order to evaluate the feasibility of OM using LQAS, the assessment team set out to answer the following questions and provide recommendations on how to improve future applications of OM: Can OM be carried out in country? Is it timely? Is it cost-effective? Is it precise? Can OM be carried out in country? The answer is that data collection can be carried out in country. In Guatemala, CIENSA successfully collected the data, cleaned it, processed it, and sent it to FANTA for analysis. PENSER performed the same tasks in Madagascar. Analysis for both surveys was done by FANTA in the United States. However, with a streamlined approach to analysis and training of local groups, this could be performed in country in the future. Is OM timely? There are a number of steps that can be taken in order to ensure that OM is a timely process. The demonstration experiences took longer than is useful for annual monitoring, but this is not unexpected for demonstrations that include elements of design and initial organization of the implementation process and analysis. Surveys in both countries took around six to seven months from initiation of fieldwork to completion of final report. In Guatemala, actual data collection took around one month; in Madagascar, the same work took seven weeks. The following tables show the breakdown of time for both Guatemala and Madagascar. Adjustments can be made to most of these steps that will reduce the overall time for the survey. 12 Assessment of GH Outcome Monitoring Pilot Guatemala Activity Time Hiring interviewers Logistics preparation Obtaining letter of support from MSPAS 1 month Interviewer training, including use of personal digital assistants (PDAs) 3 weeks Data collection in 5 departments (2 teams with 1 supervisor and vehicle each; 1 team had 4 interviewers and the other had 5) 1 month (6- or 7-day workweeks) Data cleaning and initial processing to produce CD of data to send to FANTA in US 5 days (4 days for separation of questionnaires for each age group and 1 day of data cleaning) Analysis of data in US Data analysis almost finished 1½ months Report generation Report expected at end of November, 6 months after study was initiated Madagascar Activity Time Hiring interviewers Logistics preparation 1 month Interviewer training, including use of PDAs 2 weeks Data collection in 4 supervision areas. (2 teams with 1 supervisor and vehicle each; 1 team had 4 interviewers and the other had 5) 7 weeks Data cleaning and initial processing to produce CD of data to send to FANTA in US 2 weeks Analysis of data in US Data analysis almost finished 2 months. Report generation Report expected at end of November, 7 months after study was initiated As a comparison, knowledge, practice, and coverage (KPC) surveys implemented by the Child Survival and Health Grants Program (CSHGP), which use LQAS or 30x10 cluster methodology, normally take 30 working days to complete. This includes logistics planning, questionnaire development, training of interviewers and supervisors, data collection and cleaning, data processing, analysis, report writing, and initial program decisions. These surveys include 17 standard indicators that are similar to the GH indicators, plus additional indicators from standard technical area modules. The survey is applied to mothers of children 0 to 23 months of age. A good example of a survey that was completed in close to 30 days was a survey that Helen Keller International (HKI) implemented in Niger in 2005. This survey used 30x10 cluster design but collected indicators similar to OM indicators. Questionnaire design, sampling frame determination, cluster selection, and logistics planning took 14 days. These activities were performed as part of a survey trainer’s course – KPC training of survey trainers (TOST) that was held from August 8 to 19, 2005. In-country training took two days. Fieldwork was completed in five days. The final report was finished in September 2005. In most cases, the project area for a KPC survey is smaller than a USAID Mission program area, but in some cases many more districts are included. A good example of this is Plan Cameroon’s Assessment of GH Outcome Monitoring Pilot 13 expanded impact project (2005–2010), which implemented a KPC survey in 11 districts using LQAS and made use of local organizations for data collection. KPC surveys for CSHGP can be implemented in a short amount of time because CSHGP has invested a lot of effort into developing guidance that includes standard questionnaires, sampling methodologies, and training material. This guidance could be a starting point for developing similar guidance for OM. The following are practical suggestions for speeding up the process. Most of these ideas came directly from CIENSA based on its experience with this demonstration effort in Guatemala. Other ideas are from the assessment team’s analysis of the situation. Make sure that Spanish versions (in the case of Guatemala) of questionnaires are finished and have gone through initial pretesting before training interviewers and supervisors. – This would shorten training time and reduce costs of the training by reducing total per diem and logistics expenses. Although some pretesting is needed during training to ensure that the questionnaires are adapted to the local setting, the basic instruments should be tested earlier. Develop training material before running the training session. – Because this was a demonstration experience, standard training material was not available. – Training material developed for CSHGP can be used as a reference for developing standard training for OM surveys. LQAS training manuals are also available.4 Material from these sources should be incorporated into future OM training. – CIENSA developed a PDA user’s guide that is quite useful and could be used in the future, although translation would be needed for use in non-Spanish-speaking countries. Letters of support should be obtained from the MOH before scheduling field work. – It is not safe for interviewers to go into communities without official approval of the activities. It is also important to make sure that local officials and community leaders endorse field activities. PDA programming should be improved to facilitate field work and analysis. – Improved check programs should be incorporated to enable teams to check questionnaires while still in the field. The team in Guatemala was not able to detect certain problems until it was back in Guatemala City, when it was very time-consuming and costly to obtain the missing information. – Questionnaires for the different age groups should be in separate files in the PDA. Linking the files created problems for interviewers, who had to return to previous data input screens to correct information during interviews. After interviewers corrected the earlier information, it was hard for them to find their place in the electronic input file in order to continue with the interview. 4 Valadez, J. J., W. Weiss, C. Leburg, & R. Davis. 2003. Assessing Community Health Programs: Using LQAS for Baseline Surveys and Regular Monitoring. St Albans England: Teaching-aids at Low Cost. 14 Assessment of GH Outcome Monitoring Pilot Streamline the questionnaire to reduce the number of questions. This would result in a smaller data set with fewer variables and would facilitate manipulation of the data set by statistical packages. Analysis time would be shortened. The basic questionnaire with the GH indicators could be simplified without eliminating indicators. The OM survey should concentrate on key indicators. This decision will significantly reduce the time allocated to data collection, data analysis and report writing. – CSHGP collects almost the same indicators, with fewer questions. It has been used in many countries.5 – Countries should be advised to limit the number of additional indicators that they add to the survey because it will impact on the timeliness and cost of obtaining survey results. FANTA suggested that before the next OM experience, time should be spent on improving the organization of implementation and analysis based on these demonstration experiences. This would improve the timeliness of the entire process. Is OM cost-effective? In general, OM approaches including LQAS and 30x10 cluster KPC surveys are quite cost effective. The USAID Missions in Guatemala and Madagascar feel that these studies are worth the investment and plan on funding them in the future. They were pleased with the demonstration efforts and thought that they had been a good investment. However, they would welcome any reduction in costs of future studies. In the future, it should be possible to reduce the costs of OM studies. The demonstration efforts included design and testing, which elevated their costs. These costs would not be included in future OM studies, once standardized guidance for implementing OM has been developed. Also, costs would be reduced by implementing the recommendations from the previous section on timeliness that described ways of streamlining the process. Another way to reduce costs that was suggested by implementing partners in Guatemala and Madagascar is to increase participation of partner staff in data collection. In addition to saving money, including partner staff would have benefits for improving quality of data collection and use of data for decisions. The following are advantages to partner participation: Community members are more likely to accept interviews from partner staff because they are well known in the communities where they work. Personnel from partner organizations usually know the local languages, which facilitates interviews Partners are motivated to collect quality information because they see the usefulness for the areas where they work Partners are more likely to use the information to adjust programming if they are closer to the data collection process. In addition, it is important to consider the location of training sessions. Sometimes bringing interviewers to the capital city, as was done in Guatemala, is more expensive than holding the training closer to the survey site. However, this would have to be determined for each survey. 5 The link on the Child Survival Technical Support Plus (CSTS+) Web site to the Rapid CATCH 2006 questionnaire and tabulation plan, which contain many of the GH indicators but with fewer questions, is at: http://www.childsurvival.com/kpc2000/kpc2006.cfm Assessment of GH Outcome Monitoring Pilot 15 For future OM studies it will be important to carefully plan for the costs of these studies. The KPC TOST curriculum (Session 25 of Module 1, Developing a Logistics Plan and budget) contains instructions and worksheets that facilitate this planning6 . Use of these worksheets will help future OM studies to carefully determine budget needed. Worksheets are included in the curriculum in order to determine the following: The schedule of activities Personnel needed, including trainers, supervisors and interviewers and the budget needed or each type of staff Transportation plan, including number of vehicles and drivers, fuel and maps and the budget associated with the plan Editing, printing and copying survey forms and other material (This form would not be needed if hand held data collection devises are used) Plan for computerized tabulation of results Budget for services and equipment including: computers, printers, generators, software, document copying, office space rental, and cost of food during training workshops. Costs for the assessments in Guatemala and Madagascar are higher than will be needed for future studies. The following are the costs of the demonstrations. These costs include both the population based surveys and facility based surveys that were carried out at the same time. Guatemala The total cost of the assessment including FANTA’s work of design, training and analysis was $162,216. The in-country portion of this cost was $69,267. The cost per supervision area (including FANTA’s costs) is calculated at $32,443. The local cost per supervision area (excluding FANTA’s cost) was $13,853. Madagascar The total cost of the assessment including FANTA’s work of design, training and analysis was $174,797. The in-country portion of this cost was $79,778.76. The cost per supervision area (including FANTA’s costs) is calculated at $43,699. The local cost per supervision area (excluding FANTA’s cost) was $19,944.69. Is OM precise? Outcome Monitoring using either LQAS or 30x10 cluster sampling provides coverage estimates that are sufficiently precise to measure outcomes for the majority of USAID funded programs that work at the population level. Both classic LQAS and 30x10 cluster sampling provide coverage estimates with ± 10% precision and 95% confidence levels. In other words, a coverage proportion calculated with a total sample size of 95 will have a confidence 6 The link on the CORE Web site to the KPC Training of Survey Trainers curriculum is at: http://www.coregroup.org/working_groups/kpc_training/welcome.html 16 Assessment of GH Outcome Monitoring Pilot interval that will always be less than +10%. It means that an estimate of 80% coverage proportion has a 95% chance of being within 10% of the estimate (between 70% and 90%). For coverage levels of actual data collected from OM surveys, precise confidence intervals are calculated using the following formula: Formula: P=p+/-Z*SQRT((p*q/[n/D.E.]) Where: P=the actual rate/proportion in the general population p=the survey estimate q=1-p z=the confidence level (with 95% confidence leel, z=1.96) SQRT=square root n=sample size D.E.=Design Effect (which is 1 for SRS7 and LQAS with SRS and is 2 for 30x10 cluster) In the case of the pilot studies there were two factors that had additional effects on precision: (1) the modifications to the conventional LQAS and (2) the small sample size for Madagascar. For the Pilot studies, FANTA used a modified version of LQAS. Normally, sampling in LQAS is basically Simple Random Sampling (SRS). In order to select the households, the project area is divided into at least 5 supervision areas. In each supervision area 19 households are selected randomly. The data from individual supervision areas or strata are pooled into an estimate of coverage for an entire program area. In typical applications, when all strata are pooled the result from each supervision area is weighted by the size of its population. The modification used by FANTA introduced an element of clustering. For each supervision area, seven clusters were chosen and three interviews for each age group were conducted in each cluster for a total of 21 interviews per supervision area for each questionnaire. Computer simulations demonstrated that this approach is valid when inter-cluster and intra-cluster correlation are low.8 FANTA will test this hypothesis against data from Guatemala and Madagascar in order to determine if inter-correlation and intra-correlation are low in an actual field setting. If the hypothesis turns out to be true, then one would say that this modified version of LQAS does yield precise information. If the hypothesis turns out to be false, then OM should use classic LQAS in order to obtain precise information for program management and reporting. In Madagascar, 84 interviews for each age group were conducted. This is below the usually recommended number of 95 interviews in program area. As mentioned above, the sample size of 95 is suggested in order to have a coverage proportion calculated with a confidence interval that is always less than +10%. The sample size of 84 would have a wider confidence interval (CI), which results in lower precision. Comparisons show that CIs for the Madagascar data are wider than for the Guatemala data. 7 SRS=Simple Random Sampling 8 Olives, Casey. 2006. Madagascar LQAS Simulations. Department of Biostatistics. Harvard School of Public Health. August 24, 2006. Assessment of GH Outcome Monitoring Pilot 17 Preliminary results, with CIs for key indicators, are available for both Guatemala and Madagascar. CIs for these data were calculated taking into consideration design effect (see appendix H for preliminary results). The following are a few examples of CIs for key indicators that show the effect of Madagascar’s smaller sample size: Confidence Intervals for Selected Indicators Indicator Guatemala Madagascar Estimate CI Estimate CI % newborns receiving essential newborn care 36% 29%-43% 45% 32%-59% % children ages 12-23 months fed according to a minimum standard of infant and young child feeding (IYCF) practices 65% 57%-72% 44% 32%-57% % children 12-23 months who received at least 3 DPT card verified + mother’s recall 82% 73%-90% 86% 76%-96% % children with diarrhea in the last two weeks who were given ORT 71% 62%-80% 34% 22%-45% Overall, the estimates give information that is good enough for management decisions and for looking at trends over time, although CIs are wider for some indicators, such as use of oral rehydration therapy (ORT) in Madagascar than others, such as diphtheria-pertussis-tetanus (DPT) immunizations in Guatemala. In the future, it would be best to have at least 95 interviews for each indicator in the survey. It will be important to look at inter- and intracluster correlation in order to decide whether modified LQAS yield data that are precise enough for management and reporting purposes or if classic LQAS should be used. USEFULNESS OF THE OM DATA FOR PROGRAM MANAGEMENT, DECISIONMAKING, AND REPORTING BY MISSIONS AND IN-COUNTRY PARTNERS The OM process was designed to give USAID Missions outcome-level health information to help them make decisions about programming more often than DHS, which take place every three to five years, and for areas that are specific to where USAID works. At the same time, USAID intends for this methodology to be valuable to the country and to implementing partners. USAID Missions from both countries value the information from OM surveys and want to repeat them. Partners in Guatemala and Madagascar see this information as being valuable to them and to the country. Although all these groups value this information, it is important to take a critical look at the purpose of OM, how OM information can be used, what the limitations are, and the opinions of different groups interviewed about using OM information. Purpose of OM: OM is designed as a rapid population-based assessment that can provide information on coverage of key health indicators that can be used for decisionmaking. It is important to clarify that this process is for monitoring and not exclusively for external oversight, which is not clear to all USAID Mission staff members or to implementing partners. Concentrating on the monitoring aspect permits a participatory process at all stages, including 18 Assessment of GH Outcome Monitoring Pilot design, data collection, data analysis, and decisionmaking. Members of local implementing partners could participate in the data collection if properly supervised so that they follow a standard sampling protocol. This would increase buy-in of implementing partners, increase credibility of findings with partners, and improve partners’ use of this information for decisionmaking. In addition, sustainability of the effort would be possible, since these organizations would be able to carry out the process in the future and use the OM information. In contrast, if it is decided that this process should be seen as part of external oversight, an outside firm will always be needed, and implementing partners will be less motivated to act on the findings. It will be harder to maintain sustainability in country if OM is used exclusively for external oversight. However, if a Mission finds it important to use OM as part of an external oversight process, valid information could be collected using an outside firm as was done for the two pilot efforts. How OM can be used: OM information can be used by both USAID/Guatemala and USAID/Madagascar to look at coverage levels for all indicators for the USAID program area. They can judge what technical areas need more effort or what strategic decisions should be made and implemented to improve the situation. For example, the preliminary information listed in the above table “Confidence Intervals for Selected Indicators” shows that in Madagascar, ORT use is low (34 percent) and requires added attention. For Guatemala, essential newborn care is low (36 percent) and should receive more effort or focus. If these surveys are repeated annually, the Missions will be able to see trends in these indicators and can continue to adjust programming. Guatemala will be able to further refine the analysis of its program area because the OM survey was implemented with a geographic focus in the main area of USAID Mission activity. The survey was implemented in five regions – Totonicapan, Quiche, San Marcos, Chimaltenango, and Quetzaltenango. It will be possible to find out whether each region is meeting acceptable levels in each of the indicators (this information will be available in the final report). For example, perhaps the survey finds that ORT meets targets in all regions except Totonicapan. The Mission can then concentrate attention on Totonicapan to improve ORT use instead of spreading this effort throughout the entire program area. The Madagascar survey was not set up this way, so this level of detailed information for programming will not be available. Outcome indicators complement output indicators reported in Operational Plans (OPs) and collected through the Foreign Assistance Coordination and Tracking System (FACTS). Both types of information allow USAID Missions to make management decisions. The implementing partners in Guatemala proposed an important use for OM information for them and for the country. They proposed that the MOH gather partners that work in the USAID programming area and examine the data, both coverage levels for the entire area and performance (pass/fail in meeting targets) of health districts. Then they can collectively decide how each organization can contribute to improving these indicators. USAID/Guatemala liked this idea. Another use for OM information could be to calculate the number of lives saved based on the methodology that is used by the Child Survival and Health Grants Program (CSHGP) that is Assessment of GH Outcome Monitoring Pilot 19 based on the methodology used in the Lancet series on child health. This methodology can be accessed on the Child Survival Technical Support Project + website9 Limitations of use of OM approach: The survey in Madagascar was not set up by discrete geographic regions, as was done in Guatemala, so it will not be possible to gather partners together to plan how to address health problems based on OM information. For Madagascar, it is not possible to judge if individual supervision areas are meeting or not meeting targets. Although the Mission in Madagascar was interested in using this information to look at coverage by implementing partner, the survey was not designed to accomplish this. The OM survey as implemented in Madagascar cannot be used to judge performance of individual implementers. It is more efficient to judge the performance of individual implementers using information generated by each implementing organization’s own M&E system. In Guatemala the OM survey was organized in a way that does not allow the calculation of coverage proportions at regional level. However, it is possible to determine whether or not each supervision area (region) meets its targets and to estimate coverage proportions at the aggregate level for the entire USAID program area. These surveys are not appropriate for judging the success of nationally focused efforts, such as policy reform. Again, specific project M&E systems can provide this information. Opinions of groups interviewed: The interviewing process yielded the following additional comments about the usefulness of the information and advice for future use of this information. Implementing partners in Guatemala find OM to be extremely important for the country and recommend that it be repeated annually. The following are some of their reasons: OM can help partners understand how their narrow focus contributes to the larger public health situation. OM can help partners gain a better understanding of what is needed in regions where they work. OM complements project monitoring systems. MSPAS noted that OM complements MSPAS’ regular health information system (HIS) and could help check for accuracy. USAID/Madagascar had these additional comments: OM can help tell the story of USAID achievements and be used to leverage more funding from Washington. OM can help the Mission identify technical areas that require greater attention. OM allows USAID to redirect efforts to areas of greater need. Each implementing partner has its internal HIS that could be strengthened with OM results. USAID will make funding decisions based on OM results. ASSESSING LOCAL TECHNICAL CAPACITY IN USING LQAS FOR OM 9 http://www.childsurvival.com/start.cfm 20 Assessment of GH Outcome Monitoring Pilot There is in-country capacity in both Guatemala and Madagascar for data collection for OM using modified LQAS although some technical assistance (TA) will be needed for a while. In each country, the group subcontracted by FANTA (CIENSA in Guatemala and PENSER in Madagascar) successfully collected data from the field and performed initial processing so data could be sent to FANTA for final analysis in the United States. Both groups would be capable of analyzing the data in country, provided that the analysis process is better organized and standardized and that people in the organizations are trained in this process. They would also need access to limited TA in order to answer any questions during the analysis. In both Guatemala and Madagascar, some implementing partners have experience with population surveys using conventional LQAS. Most implementing partners, if given adequate training, could implement OM surveys using modified or conventional LQAS. However, most implementing partners lack sufficient human and financial resources to do this. In addition to implementing partners and CIENSA, the Nutrition Institute for Central America and Panama was mentioned as a good M&E resource for implementing OM surveys in Guatemala. In Madagascar, each implementing NGO interviewed has an M&E unit with specialists (statistician, medical doctors, engineers, and demographers). Partners have routine M&E systems that include monthly, quarterly and annual reports; meeting reports; and supervision checklists. All partners have computers and appropriate software packages. Partners have used conventional LQAS but not modified LQAS. For example, the MCDI project director received KPC training in North Carolina in 2000 and has used the LQAS methodology both for household and facility surveys and trained other PVOs. Most partners in Madagascar need training in LQAS, use of PDAs, and use of results for decisionmaking. The same training would be useful for Guatemala. One observation from the assessment in Guatemala is that partners are currently implementing surveys using a variety of techniques. Some are using conventional LQAS, and others are using 30x10 cluster. One group is using a very different design in which more than one child per household is interviewed. This is similar to the approach that DHS uses, but it complicates the analysis because children from the same household will receive treatment that is more similar than when compared with children from different households. In addition, this organization implements surveys that use an approach to taking into consideration different regional population sizes that is different (but valid) from that used in the modified LQAS, conventional LQAS, or 30x10 cluster sampling. The important message from these observations is that partners will need training on OM with modified or conventional LQAS in order to understand the results and in order to use this methodology themselves. One commonly shared impression is that the OM results should be disseminated, and implementing partners and the MOH should be encouraged to use the results. It should be clear who can, and should, use these results and for what purpose. Characteristics that local organizations need to have in order to take over the OM data collection and analysis include: a) technical skills and knowledge to better understand LQAS and its application; b) facilitation skills for partners’ orientation before the survey and result dissemination; c) capacity to identify and communicate actionable findings for improvement; d) expertise in operational research; e) strong connections and public relations with the government and other stakeholders; and f) good experience in public health. Assessment of GH Outcome Monitoring Pilot 21 STRATEGIES FOR TRANSFERRING THE OM APPROACH TO MISSIONS AND IN￾COUNTRY PARTNERS AND FOR EXPANSION TO OTHER MISSIONS In order to ensure that the Guatemala and Madagascar USAID Missions and in-country partners are able to continue the OM approach and that other Missions can make use of this approach, it is necessary to set up an organized system of guidance and limited central TA. It will be necessary to invest some time up front to organize these supports and limited time afterward to maintain them. The following should be developed: Standardized questionnaire for GH indicators: – This should be a streamlined version of the one that was used in the pilot studies. It will have to be updated periodically, based on feedback from field use and changes in international thinking about measurements for the technical areas included in OM. Standardized and detailed instructions for implementing modified LQAS or conventional LQAS, depending on the methodology that GH chooses for OM: – This should include instructions to maintain a geographical approach to this survey, as was used in Guatemala, as opposed to an implementing partner approach that was tried in Madagascar. – This should include instructions on how to use information from LQAS at both aggregate and supervision area levels. Standardized training material, which can be adapted from the CORE Group’s KPC TOST curriculum and from the LQAS training manuals. Guidance about the philosophy of OM, which should include the following: – An explanation about the purpose of OM, which is for monitoring and not exclusively for external oversight – An explanation of how OM relates to the output indicators that are now being reported to the Office of Foreign Assistance and how both are useful for management decisions – Suggestions about how OM information can be used by local partners to adjust programs to contribute to improving health indicators in the regions where they work – An explanation of how OM complements other information, such as information from DHS, Multiple Indicator Cluster Surveys (MICS), or routine information from the MOH Guidance with suggestions on how OM can build M&E capacity in country, including the importance of using a participatory approach to the OM process Guidance with suggestions on how OM can be used by partners in country to make decisions about programs: – This guidance could center on the idea proposed in Guatemala of gathering partners together for a discussion of the results and what each partner can do to help solve health problems uncovered by the survey. 22 Assessment of GH Outcome Monitoring Pilot Limited technical support at a central level to help Missions with implementation questions and to ensure that Missions understand adjustments made to the survey due to changes in technical areas Assessment of GH Outcome Monitoring Pilot 23 III. RECOMMENDATIONS All the recommendations from this assessment can be implemented. OM is useful and can be a simpler process if the recommendations are taken into consideration. Review and communicate to Missions and partners the purpose of OM: – Emphasize monitoring – Emphasize that this should be a simplified process Discuss and communicate to Missions that the best use of OM is to review the health situation of discrete geographic areas: – This methodology as implemented in Madagascar and Guatemala is not well suited to judge the success of individual partners. For this, it is better to use a project’s own M&E system. The geographic focus helps Missions make management decisions while helping partners see their role in improving the overall health situation of the population. Develop systematic training plans, including planning for logistics management: – Training material from the CORE Group (KPC TOST)10 is useful as a starting point. – LQAS manuals are also available.11 Streamline the questionnaire to reduce the number of questions: – This would result in a smaller data set with fewer variables and would facilitate manipulation of the data set by statistical packages. Analysis time would be shortened. In addition, data collection time would be reduced. The basic questionnaire with the GH indicators could be simplified without eliminating indicators. – CSHGP collects almost the same indicators with fewer questions. Its questionnaire has been used in many countries. – Countries should be advised to limit the number of additional indicators that they add to the survey because it will impact on the timeliness and cost of obtaining survey results. Reduce the number of questionnaires by reducing the number of age groups in the survey: – It is important to remember that OM should be a rapid monitoring tool. With this in mind, it would be possible to ask FP/RH questions to mothers of children 0 to 11 months old or mothers of children 12 to 23 months. This would provide good information about FP and RH in the program area without making the survey too complex. – Also questions asked about children 0 to 59 months could be asked about children 0 to 23 months. The only caveat is that countries may want to report on malaria 10 The link on the CORE Web site to the KPC Training of Survey Trainers curriculum is at: http://www.coregroup.org/working_groups/kpc_training/welcome.html 11 The link on the CORE Web site to the LQAS training manual is at: http://www.coregroup.org/working_groups/lqas_train.html 24 Assessment of GH Outcome Monitoring Pilot information for 0 to 59 months to be consistent with Roll Back Malaria, and their opinion should be taken into consideration. – These changes would reduce data collection and analysis time. Decide whether to promote use of modified LQAS or conventional LQAS for outcome monitoring: – FANTA plans to conduct analyses of intercluster correlation and intracluster correlation with Guatemala and Madagascar data to complement the computer simulation already performed. The results of this analysis should help determine the validity of the modified approach. Intercluster correlation refers to the degree with which interview points (mothers) are related to other interview points between clusters. Intracluster correlation refers to the degree with which interview points are related to other interview points within a cluster. – Another consideration for this decision is the need to explain the validity of modified LQAS to stakeholders. – A careful analysis of data collection time saved and cost by modified LQAS vs. conventional LQAS as actually implemented would be a useful input into the decision of whether or not to use modified LQAS. Improve the involvement of MOH and partner organizations: – Partners are interested in being involved at all stages of the process including design, data collection, data analysis, and decisionmaking. – Members of local implementing partners could participate in the data collection if properly supervised so that they follow a standard sampling protocol. This would increase buy-in of implementing partners, increase credibility of findings with partners, and improve partners’ use of this information for decisionmaking. – In addition, sustainability of the effort would be possible, because these organizations would be able to carry out the process in the future. – In contrast, if it is decided that this process should be seen as part of external oversight, an outside firm will always be needed and implementing partners may be less motivated to act on the findings. Ensure that survey results are produced in a short period of time: – Before implementing OM again, spend time on improving the organization of implementation and analysis based on pilot experiences and considering experiences of other organizations. Make sure that partners and communities receive feedback on the results and understand how to interpret the survey findings: – A dissemination workshop should be held as part of this process. The following actions would help transfer OM capacity to Missions and in-country partners and expand OM to other Missions: – Develop tools and guidance based on the recommendations from pilot efforts. – Develop guidance that explains that the purpose of OM is monitoring and not just external oversight. – Maintain limited technical support at a central level for questions about survey implementation and to ensure that Missions understand adjustments made to the survey due to changes in technical areas. Assessment of GH Outcome Monitoring Pilot 25 One important finding from the assessment of the OM pilots is that both the Guatemala and Madagascar Missions value the information on outcome indicators as collected by these pilots. If OM is rolled out to other Missions, it would be possible for the same information to be collected by using classic LQAS (without clustering) or cluster sampling. If Missions are given a choice of other methodologies, the following should be taken into consideration: – For cluster sampling, in order to obtain information about sub-areas, each sub￾area would require a separate survey. These separate surveys could then be combined to obtain a coverage estimation for the entire program area. – If Missions are to be given a choice of sampling methodologies, they would need detailed guidance for each methodology. – Choice of methodologies should be limited because it would be too difficult to monitor the quality of OM surveys implemented if Missions used their own methodologies. Classic LQAS and 30x10 cluster sampling are two of the more highly used sampling methods to assess public health programs and should be considered by Missions for OM. Classic LQAS Interest in applying LQAS to health assessments has been growing and gaining popularity since the mid-1980s. In September 2006, the World Health Organization (WHO) and the World Bank undertook a global review of the use of LQAS methodology and found more than 800 applications worldwide, thus providing an ample view on the current status of LQAS.12 In Bangladesh, ICCDR/B explored the validity of this methodology by carrying out a survey using LQAS in the Matlab catchment area (2007) and found no difference between the LQAS survey information and that obtained through ICCDR/B’s census-based system.13 The ICCDR/B concluded that the LQAS method “… for many practical purposes, [with] a sample size of 19 should serve the purpose for programme managers.” This result is particularly important since it is the first time that LQAS has been compared with a census and therefore establishes the validity of the results. With respect to LQAS costs, the most comprehensive study was carried out by Christophe Grundmann14 in 2002 under a contract financed by USAID. His work compared a cluster sample with multiple LQAS applications. His first conclusion was that LQAS costs should not be compared with other surveys without first emphasizing that LQAS results also support program management. Therefore, while the costs of other surveys typically are for M&E only, LQAS has 12 Robertson, S.E. & J.J. Valadez. 2006. “Global review of health care surveys using lot quality assurance sampling (LQAS), 1984-2004.” Social Science and Medicine. 63:1648-1660. 13 Abbas Bhuiya, S.M.A.H., Nikhil Roy, & P. Kim Streatfield. 2007. “Performance of the Lot Quality Assurance Sampling Method Compared to Surveillance for Identifying Inadequately-performing Areas in Matlab, Bangladesh.” Journal of Health Population and Nutrition 25(1): 37-46. 14 Grundmann, Christophe. 2002. “The Costs of Using LQAS for Project Management, Monitoring and Evaluation.” NGO Networks for Health Project. 26 Assessment of GH Outcome Monitoring Pilot a management function, so some of its costs should not be attributed to M&E but instead should be attributed to program management. Grundmann also showed that once LQAS is used recurrently, economies of scale begin to take hold and the costs decrease considerably. Classic LQAS is well proven and should be recommended for OM as an effective, feasible, and affordable approach to data collection that can provide very useful information to both USAID Missions and other engaged agencies. 30x10 Cluster Methodology This document mentions the 30x10 cluster methodology that is used by USAID’s CSHGP. It is based on 30x7 cluster methodology used by the Expanded Program on Immunization since the 1980s. The KPC survey was initially developed by the Johns Hopkins University Child Survival Support Project in 1990 and has been used by PVO grantees since 1991. Initially the KPC used 30x10 cluster sampling. PVO grantees apply this survey at the baseline, mid-term, and final stages. It has been used in more than 50 countries in Africa, Asia/Near East, Latin America, Europe, and Eurasia. It has been implemented in both rural and urban areas. The KPC collects information on a variety of technical areas: immunization, control of diarrheal diseases, nutrition, micronutrients, breastfeeding, birth spacing/FP, acute respiratory infection/pneumonia, malaria, maternal and newborn care, STIs, HIV, and AIDS. It could be offered to Missions as an alternative sampling methodology for OM. More recently, KPC surveys are being implemented using LQAS (classic LQAS). The same questions are used for LQAS as for 30x10 cluster surveys. The results are compatible. PVO grantees can choose either sampling methodology. For the current group of 59 active grants, 14 used LQAS at baseline, and 45 used 30x10 cluster sampling methodology.15 15 Child Survival and Health Program database maintained by Macro International, CSTS+. Assessment of GH Outcome Monitoring Pilot 27 IV. APPENDICES APPENDIX A: SCOPE OF WORK Background: With the advent of foreign assistance reform, increasing emphasis has been placed on the collection of output measures that can be gathered annually from partners and that are attributable to USG investments. While output indicators are important for short-term budgeting and reporting, they are often poorly correlated with outcomes because they do not capture important processes that transform outputs into the desired outcomes. USAID has traditionally measured progress in the health sector at the population level using internationally recognized outcome and impact measures, such as contraceptive prevalence rates, immunization rates, the proportion of births with attended deliveries, use of bednets, etc. At the national level, progress against these indicators can rarely be attributed to any single donor. In addition, it may not be possible to reliably measure change in some of these indicators on an annual basis. GH is interested in the feasibility and usefulness of collecting outcome measures more frequently than is currently done through the Demographic and Health Survey, which is typically fielded every 3-5 years, and at a level more directly related to USG assistance, i.e. for USAID-assisted areas rather than at the national level. The rationale for more frequent and directly related to outcome data collection is two-fold: 1. First and foremost, such data are needed to inform and improve on-going program implementation. Many missions have expressed interest in lower-cost survey methodologies that can provide more timely and relevant information for planning and budgeting. To the extent that it is possible to define standardized, common outcome indicators that can be collected at regular intervals in USAID- assisted program areas between national DHS surveys, it will be possible to both adjust program implementation approaches at the country level to improve performance and to more easily identify best practices across countries. 2. The output indicators and data reported in Operational Plans (OP) and collected through the Foreign Assistance Coordination and Tracking System (FACTS) need to be complemented with outcome indicators and data. By having both levels of data available, it will be easier to identify the most promising program approaches and the output measures that are the best proxy indicators of desired outcomes. In September 2006, GH sponsored a one-day expert consultation to explore the feasibility and cost￾effectiveness of collecting outcome data on an annual or bi-annual basis. The findings from that meeting suggest that there are existing methodologies that can be used to estimate key outcomes at the population￾level in USAID-assisted areas on an annual basis at a reasonable cost. GH decided to pilot the use of the Lot Quality Assessment Survey (LQAS) approach for this purpose principally because of its perceived cost advantages (due to smaller sample sizes) and because it can generate both program-wide summary statistics as well as lot-specific information. Like the other methods considered, LQAS is also a known and proven approach to data collection. In December 2006, GH initiated a discussion with the FANTA Project, managed by the Academy for Educational Development, regarding the collection of FP/RH, MCH, and malaria outcome data using a modified LQAS approach on a trial basis in two countries, Guatemala and Madagascar. Both of these countries were already using LQAS to collect other information. GH Outcome Monitoring Pilot field work in Madagascar is expected to take place in June and July and field work in Guatemala is expected to take place from mid-July to end of August, 2007. Preliminary data from Madagascar will be available at the end of August and data from Guatemala will be available by the end of September 2007. Purpose of the Assessment: The purpose of this assessment is to: 28 Assessment of GH Outcome Monitoring Pilot 1. Assess the feasibility of conducting LQAS-based Outcome Monitoring (OM) in these settings annually or biannually; 2. Assess the usefulness of the data for program management, decision making, and reporting by missions and in-country partners; 3. Assess local capacity issues related to the transfer of the approach to Missions and in-country partners over the medium term. 4. Assess USAID mission and AID/W bureau interest in annual or biannual outcome data collection. The results of the assessment will be used as input to decisions about 1) whether LQAS is a feasible and appropriate methodology to use for annual or biannual monitoring of outcome indicators; 2) how to maximize the usefulness of the results for Mission programming; and 3) how to transfer capacity to implement the OM approach to Missions and in-country partners. Statement of Work: The questions to be answered by this assessment flow from the four purposes above: 1. Assess the feasibility of conducting LQAS-based OM in these settings annually or biannually. a) What was the experience using the LQAS OM approach to collect data on selected outcome indicators in FP/RH, MCH, and malaria in Guatemala and Madagascar? b) How feasible does it appear to be to use the LQAS OM approach to collect such data annually or bi-annually? c) How valid does it appear to be to collect the GH-specified outcome indicators in this manner? d) What did it cost to collect the data in each country? What factors influenced the cost? What is it likely to cost in other countries? e) What modifications to the approach are needed, if any, to make it easier to implement? What additional tools/instructions/guides (e.g., common indicator definitions, tabulation plans, resources, etc), need to be developed, if any, to make the data collection go more smoothly? f) What modifications could make it less costly?(For d) and f), please include your assumptions as well as what is and is not included in your cost estimates.) 2. Assess the usefulness of the data for program management, decision making, and reporting. (Some of these questions may need to be adjusted to focus on potential usefulness if there has not been sufficient opportunity to use the results). a) Were the GH-defined outcome indicators relevant and useful for program management, decision￾making, and/or reporting by the mission and/or in-country partners? What changes might be needed to increase relevance or usefulness? b) How many and which additional indicators were collected beyond the GH-defined outcome indicators? c) What did USAID/Guatemala and USAID/Madagascar and in-country partners hope to be able to do with the data? d) What are they able to do with the data? e) If the data have already been used in program management, decision-making or reporting, how were they used? What decisions did they influence? f) What other ways might the data be used? g) What modifications to the approach are needed, if any, to make the resulting information more useful for program management, decision making, or reporting? 3) Assess local capacity issues related to medium-term transfer of the approach to USAID missions and in-country partners. a. What local partners were used to implement the LQAS OM approach in the two countries and what were their roles in the process? b. What is the minimum skill set needed in-country to conduct this kind of data collection? c. What is the minimum skill set needed in-country to make good use of the resulting information? Assessment of GH Outcome Monitoring Pilot 29 d. What local organizations (in a generic sense) are best-positioned to assist with this kind of data collection? e. How can the LQAS OM process be used effectively and cost-efficiently over the medium term to build local capacity to undertake LQAS while at the same time having the data available in a timely manner? f. What other kinds of capacity building, perhaps not linked to the LQAS OM approach directly, are needed to help institutionalize Mission and local capacity to undertake annual or biannual outcome monitoring ? 4. Assess selected USAID mission and AID/W bureau interest in annual or biannual outcome data collection.16 a. What outcome information do health officers want on an annual/bi-annual basis? b. If missions are collecting outcome data annually or biannually already, how are they doing so? c. How many missions are already using LQAS? How well suited is LQAS to collecting the desired information? d. How many missions with programs in FP/RH, MCH, and/or malaria express interest in annual or biannual outcome data collection? How interested are AID/W bureaus? e. What set of criteria are missions using to decide whether they are interested? f. How much of the cost of fielding annual or biannual outcome data collection are interested missions willing to pay? g. What reasons are given for not being interested? h. What specialized TA, if any, do missions need to collect outcome data annually or biannually (e.g., sampling, setting benchmarks, implementation, interpretation of results, report writing)? i. Would missions prefer to access such TA through a locally-contracted mechanism or a GH-based project? Methods and Procedures Data Sources. The FANTA project is a principal data source for this assessment, including the staff who have been involved in designing and implementing the OM activities in Madagascar and Guatemala, as well as information in the project files. The data for this assessment will come from documents provided by FANTA as well as interviews with key informants. FANTA will arrange to present an overview of the GH Outcome Monitoring project and preliminary results at the beginning of this assessment to the team and GH team involved in this pilot. Other key informants include GH staff involved in conceptualizing the trials; USAID mission staff in Madagascar and Guatemala; the local firms contracted for data collection in Madagascar and Guatemala; staff of implementing partners in Madagascar and Guatemala; USAID PHN staff in other countries; senior GH management who will need to approve the design of any new central project; staff of the Strategic Information unit at State/F who are involved with the OP and FACTS processes; and selected CAs and donors about their experience using LQAS. Methods of Data Collection. A range of data collection methods will be used. The assessment team will undertake a desk review of relevant FANTA project files and other background materials, such as the report from the September 2006 expert consultation on outcome monitoring and the Madagascar and Guatemala Performance Management Plans. Lists of the outcome indicators of interest to GH included in the pilots and of key documents are attached. The team will interview DC-based informants in person or by phone and mission and local staff in Madagascar and Guatemala by e-mail and/or phone. With participation by the USAID staff involved in the pilot, the team will also design and administer an electronic questionnaire to mission health officers to assess interest in the OM approach among other 16 Subsequent to submission of the SOW to GHTech, USAID, GHTech, and the assessment team agreed to modify Purpose #4 to read: “If the Assessment Team determines the LQAS OM approach is useful to USAID, what does USAID need to do in order to “roll out” this approach to other and/or every Mission? What mechanisms must be in place prior to “roll out”? What actions are required in order to expand the application of the LQAS OM approach?” 30 Assessment of GH Outcome Monitoring Pilot USAID missions. Field visits of approximately one week to Madagascar and Guatemala may also be necessary, principally to interview counterparts and to assess local capacity issues Duration/Timing. Ideally, the assessment will begin August 2007 and be complete by mid-October 2007. Team Composition and Size: We anticipate a two person assessment team and a total level of effort of 60 days, distributed as appropriate across the two consultants. In combination, the two consultants should have the following qualifications and skills: experience in rapid outcome data collection and knowledge of sampling and statistics; familiarity with USAID health programming at the field level; familiarity with key output, outcome, and impact indicators used by USAID in the health sector; experience in assessing capacity building constraints and opportunities; ability to speak French and Spanish; willingness and availability to travel for one week; demonstrated ability to solicit and synthesize information and to convey it effectively and concisely in writing. One team member should be the designated team leader. Deliverables: A draft report is due to GH within 25 working days. GH staff will review the draft and provide comments to the team within five working days. A final report is due at the end of the 30 working-day assignment and no later than October 15, 2007. The report should be no more than 20 pages in length, exclusive of annexes. Annexes may include data tables, the questionnaire administered to mission health officers, a list of key informants, and a bibliography. The document should be submitted electronically and in five hard copies. The assessment team should also plan to debrief both FANTA and GH at the conclusion of the assignment. Funding and Logistical Support: The assessment will be funded with GH core funds through GH-Tech. GH and FANTA will provide the names of key informants and necessary background documents. GH￾Tech will arrange country clearances and in-country logistics (e.g., hotel, transportation) for field visits. Outcome Indicators of Interest to GH Included in the Pilots Malaria 1. % of household with a child(ren) under 5 with at least one ITN 2. % of children under five years of age in malaria-risk areas reported as sleeping under ITN the previous night 3. % of women who received two or more doses of SP for IPT for malaria during their last pregnancy 4. % of children under 5 years of age with fever in last 2 weeks who received antimalarial treatment within 24 hours from onset of fever Maternal and Child Health 5. % of women who gave birth who had a postpartum visit within 3 days 6. % of newborns receiving essential newborn care 7. % of women seen at ANC at least 4 times during their last pregnancy with a live birth 8. % of birth attended by a doctor, nurse or trained midwife (excludes traditional birth attendants) 9. % of children between 12-23 months of age who received their third does of DTP by age 12 months 10. % of children age 12-23 months receiving a vitamin A supplement during the last six months before the survey 11. % of children under age five who are more than 2 SD below the median weight for that age 12. % of infants aged less than 6 months who were exclusively breast-fed in the past 24 hours 13. % of children aged 0-59 months with diarrhea in the past 2 weeks who were treated with ORS 14. % of children aged 0-59 months with chest-related cough and fast and/or difficult breathing in the last 2 weeks who were taken to an appropriate health provider 15. % of children ages 12-23 months fed according to a minimum standard of infant and young feeding practices Family Planning/Reproductive Health 16. % of women of reproductive age and sexually active using, or whose partner is using, a modern method of contraception Assessment of GH Outcome Monitoring Pilot 31 17. % of need satisfied by modern method of family planning 18. % of women of reproductive age stating their desire to space birth intervals 36 months or longer, or to limit births Documents for Assessment of the GH Outcome Monitoring Pilot in Madagascar and Guatemala FANTA project Files and Documents 1. Description of the sampling methodology in Madagascar and Guatemala 2. List of Indicators (GH Outcome Monitoring and Mission Monitoring) – Madagascar and Guatemala 3. Questionnaires and software for Madagascar 4. Questionnaires and software for Guatemala 5. Sub-agreement with PENSER in Madagascar 6. Sub-agreements with CIENSA in Guatemala 7. Enumerator (PENSER) training workshop in Madagascar 8. Enumerator (CIENSA) training workshop in Guatemala 9. Cost of the GH Outcome Monitoring Pilot in Madagascar and Guatemala 10. Draft tabulations for Madagascar (end of August) and Guatemala (1st week of September) 11. Preliminary Reports for Madagascar (end of September) and Guatemala (end of October) Background Documents 12. USAID Expert Meeting September 26, 2006 – Annual Monitoring of Health Outcome Indicators: Assessment of Alternative Data Collection Methodologies: - Summary Report, - Agenda, Presentations and Participants list. 13. December 10, 2006 e-mail Chung to GH Team: attachments - List for Layers 12-20-2006.doc - Layers for outcome indicators 12-20-2006.doc 14. January 29, 2007 e-mail Chung to GH team: attachments - Issues- Indicators for outcome monitoring 1-29-07.doc - Comments on outcome indicator 1-29-07.doc - Summary of Indicators 1-29-07.doc 15. March 16, 2007 e-mail Chung to GH Team: attachments - List of outcome monitoring indicators 3-15-07 - Outcome monitoring questionnaire 3.15.07.doc - Source - outcome monitoring Qs 3-15-07.doc - FP indicators sampling algorithm 3-15-07.doc 16. April 12, 2007 e-mail Chung to GH team – attachments - GH outcome monitoring indicators- final 4-11-07 - Qs for outcome monitoring indicators – 4-11-07.doc - TOR for Madagascar External Firm 4-11-07 doc. - TOR for bidders – Guatemala External Firm.doc 17. FANTA Guatemala Trip Reports (March 19-30, May 14-25, July 1 to 6, 2007) 18. FANTA Madagascar Trip Report (April 20 - May 12, 2007) 19. Madagascar – Quality of Health Services in the USAID Mission Target Areas – Results from 2006 survey 20. Layers for Title II programs - 1 page description of Layers - a sample letter by the Mission to PVOs - PVO response to the Mission - draft Layers Manual 32 Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot 33 APPENDIX B: INTERVIEW GUIDES FOR GUATEMALA (Slightly modified from Madagascar guides) GUIDELINE FOR INTERVIEWING USAID MISSION Assessment of the GH Outcome Monitoring Pilot in Madagascar and Guatemala IDENTIFICATION Date: ______ /______ /______ Day / Mo / Year Interviewer’s Name: __________________________________________ Name of Respondent: _______________________________________________ Position: _________________________________________________________ USAID Mission (Country): _____________________________________________________ USAID/Mission health portfolio (PHN Officer) Please give me an overall description of USAID’s MCH and FP programming in country, including the following information: Names of programs Implementing partners Intervention areas Location Target population Goals and objectives Briefly tell me about a few major challenges faced by these programs Mission Monitoring System and Relevance of Annual Outcome Indicators Please tell me how you currently monitor progress of the health program: Operational Plan, Strategic Plan (Performance Management Plan, Results Framework Description) and F. What information from the recent annual OM exercise will help with this process? What did it tell you about the overall health portfolio? What did it tell you about specific geographic areas or projects? Have you or will you make any changes to the program as a result of this information? Give examples. Will you allocate funds differently because of this information? 34 Assessment of GH Outcome Monitoring Pilot Relevance of GH Indicators to Mission Monitoring; Relevance of Mission-Added Indicators (Show the list of GH indicators) For each GH-defined indicator: – Is this indicator relevant to the Mission? Why? Table 1: GH-Defined Outcome Monitoring Indicators Maternal and Child Health (MCH) 5. % of women who gave birth who had a PP visit within 3 days 6. % of newborns receiving essential newborn care 7. % of women seen at ANC at least 4 times during their last pregnancy with a live birth 8. % of births attended by a doctor, nurse, or trained midwife (excludes traditional birth attendants) 9. % of children between 12 and 23 months of age who received their third dose of DPT by age 12 months 10. % of children ages 12-23 months receiving a vitamin A supplement during the last six months before the survey 11. % of children under age 5 who are more than 2 SD below the median weight for that age 12. % of infants aged less than 6 months who were exclusively breastfed in the past 24 hours 13. % of children ages 0-59 months with diarrhea in the past 2 weeks who were treated with ORS 14. % of children ages 0-59 months with chest-related cough and fast and/or difficult breathing in the last 2 weeks who were taken to an appropriate health provider 15. % of children ages 12-23 months fed according to a minimum standard of infant and young child feeding practices Family Planning/Reproductive Health (FP/RH) 16. % of women of reproductive age and sexually active using, or whose partner is using, a modern method of contraception 17. % of need satisfied by modern method of family planning 18. % of women of reproductive age stating their desire to space birth intervals 36 months or longer, or to limit births – Relevant indicators to the Mission? (Write indicator number) – Is it useful to monitor on an annual basis or would a different time frame be more useful? (Write indicator number) – Were there any difficulties collecting or analyzing this indicator? (Write indicator number) – Do you have any comments or suggestions about these indicators? For each Mission-defined indicator: – Is this indicator relevant to the Mission? Why? – Is it useful to monitor on an annual basis or would a different time frame be more useful? – Were there any difficulties collecting or analyzing this indicator? – Do you have any comments or suggestions about these indicators? Relevance of Annual OM for Implementing Partners How do/will you use this information? How do you see implementing partners and MOH using this information? How do you see annual OM contributing to the strategic planning process and to the OP process and to the reporting process? Advantages and Disadvantages of Annual OM for the Mission and for Implementing Partners What role do you see annual OM in the future for – The Mission? – Implementing partners? Assessment of GH Outcome Monitoring Pilot 35 Is it useful for USAID Mission/your implementing agencies? Do you have any recommendations to improve the process? OUTCOME INDICATORS What was your role/responsibility during the application of OM approach? Did the mission participate in the process of selecting survey indicators? How? Were your indicators assessed in the OM survey? Did you include all the indicators that you need? What other indicators do you need? RESULTS OF OUTCOME INDICATORS Has the Mission received the OM survey results? When? OM Approach What are the main benefits and challenges of the OM approach for the Mission? What modifications to the approach are needed to make it easier to implement and less costly? What modifications to the approach are needed, if any, to make the resulting information more useful for program management, decisionmaking, or reporting? What are the characteristics of the best-suited organization for this type of data collection and analysis? What are your recommendations for expanding the use of the OM approach? FUTURE APPLICATIONS OF OUTCOME MONITORING APPROACH Are you interested in annual or biannual outcome indicator results? Would you be willing to spend Mission funds for this each year? Or, how often? What cost is the Mission willing to afford to get outcome indicator annual results? How can capacity be transferred to implement the OM approach to Missions and in-country partners? What was the total cost of implementing the OM approach in Guatemala? 36 Assessment of GH Outcome Monitoring Pilot GUIDELINE FOR INTERVIEWING CIENSA STAFF Assessment of the GH Outcome Monitoring Pilot in Madagascar and Guatemala IDENTIFICATION Date: ______ /______ /______ Day / Mo / Year Interviewer’s Name: __________________________________________ Name of Respondent: _______________________________________________ Position: _________________________________________________________ Organization: _____________________________________________________ Country: _________________________________________________________ OUTCOME INDICATORS What are the advantages and disadvantages of using the OM approach to collect FP/RH, MCH, and malaria outcome indicators annually? Why? What indicators can be annually collected using this approach? Which can’t? QUESTIONNAIRES What is your opinion about the survey questionnaires used in this process? Did the questionnaires work well? What can be improved? OM TRAINING Who participated in the OM training? What do you think about the training materials used? What can be done to improve the OM training? DATA COLLECTION Who collected the data in each supervision area? How many people in each SA? What criteria were used to select the data collection teams? On average, how much time did you spend to collect the data in each SA? What went well and what was a challenge during data collection? Did the interviewers/supervisors have any logistical problems during data collection? What challenges did the interviewers have to randomly select households and respondents? Assessment of GH Outcome Monitoring Pilot 37 How did you select the clusters in each supervision area? What was the approach to select houses and respondents in each location? What challenges did you have? What challenges did the interviewers have to conduct the interviews? How were the data collection teams organized? What mode of transportation did they use to travel within the supervision area? May I see the data collection plans? How was the experience using Personal Digital Assistants (PDAs)? What kind of problems did the interviewers have to use PDAs? What are the advantages and disadvantages of using PDAs? What is the cost of each PDA? Was there local expertise to collect the data? If you find this expertise locally, what would be the cost as compared to bringing them from the capital or other city? What were the profile/qualifications of interviewers and supervisors during the OM survey? Were any local partners involved in data collection? How were they involved? What was the total cost of data collection? What factors influenced the cost of data collection? What additional tools/guides/resources need to be developed, if any, to facilitate the data collection process? How could data collection be improved? DATA ENTRY AND ANALYSIS How much time after data collection was required to clean and package the final data and produce the CD ROM? What worked well and what difficulties did you have in this process? How can this process be improved? LQAS OM Based on your experience, what are the main benefits and challenges of using OM approach? What modifications to the approach are needed to make it easier to implement? What modifications to the approach are needed to make it less costly? What modifications to the approach are needed, if any, to make the resulting information more useful for program management, decisionmaking, or reporting? How can OM surveys be designed to provide information by geographical area (supervision area)? Cost: If this study were to be repeated again in country with only minimal help from FANTA, how much would it cost (planning; questionnaire design; training; data collection; data cleaning and entry; analysis; use of results and dissemination)? 38 Assessment of GH Outcome Monitoring Pilot GUIDELINE FOR INTERVIEWING IMPLEMENTING PARTNERS Assessment of the GH Outcome Monitoring Pilot in Madagascar and Guatemala IDENTIFICATION Date: ______ /______ /______ Day / Mo / Year Interviewer’s Name: __________________________________________ Name of Respondent: _______________________________________________ Position: _________________________________________________________ Organization: _____________________________________________________ Country: _________________________________________________________ Organization Identification/Project Description: Type of organization (institute, NGO) Please give me an overall description of the USAID-funded project(s) – Names of projects – Partner organizations – Intervention areas – Location/target population Briefly tell me about a few major challenges faced by these projects USAID-Funded Project M&E System Please tell me how you currently monitor progress of the project. What is the experience of your organization with M&E (collecting and analyzing data)? Staffing (positions and expertise) Equipment Could you please give me a copy of your project indicators? What specific knowledge do you have to use OM approach? Assessment of GH Outcome Monitoring Pilot 39 Relevance of GH indicators to Project Monitoring (Show the list of GH indicators) For each GH defined indicator: – Is this indicator relevant to the project M&E system? Why? Table 1: GH-Defined Outcome Monitoring Indicators Maternal and Child Health (MCH) 5. % of women who gave birth who had a PP visit within 3 days 6. % of newborns receiving essential newborn care 7. % of women seen at ANC at least 4 times during their last pregnancy with a live birth 8. % of births attended by a doctor, nurse, or trained midwife (excludes traditional birth attendants) 9. % of children between 12 and 23 months of age who received their third dose of DPT by age 12 months 10. % of children ages 12-23 months receiving a vitamin A supplement during the last six months before the survey 11. % of children under age 5 who are more than 2 SD below the median weight for that age 12. % of infants aged less than 6 months who were exclusively breastfed in the past 24 hours 13. % of children ages 0-59 months with diarrhea in the past 2 weeks who were treated with ORS 14. % of children ages 0-59 months with chest-related cough and fast and/or difficult breathing in the last 2 weeks who were taken to an appropriate health provider 15. % of children ages 12-23 months fed according to a minimum standard of infant and young child feeding practices. Family Planning/Reproductive Health (FP/RH) 16. % of women of reproductive age and sexually active using, or whose partner is using, a modern method of contraception 17. % of need satisfied by modern method of family planning 18. % of women of reproductive age stating their desire to space birth intervals 36 months or longer, or to limit births – Relevant indicators to the projects? (Write indicator number) – Is it useful to monitor on an annual basis or would a different time frame be more useful? (Write indicator number) – Do you have any comments or suggestions about these indicators? Advantages and disadvantages of USAID/Mission annual OM for the project What role do you see USAID/Mission annual OM in the future for the project? Do you have any recommendations to improve the process? ROLE IN PAST OM SURVEY Did you or your organization participate in the recent application of the OM approach with FANTA? What was your role/responsibility during the recent application of OM approach? (selection of indicators, preparation of questionnaires, data collection, data entry, data analysis, dissemination, use of results, others) What do you think about the recent OM survey (the way it was implemented)? What could have been done differently? Why? 40 Assessment of GH Outcome Monitoring Pilot OUTCOME INDICATORS Did your organization participate in the process of selecting survey indicators? Were your program indicators included in this survey? Did you include all the indicators that you need? Can you identify program effectiveness with the indicators that you collect with this survey? How? RESULTS OF OM INDICATORS How can your organization use the OM results? (Program management? Reporting? Planning?) Has your organization received the recent OM results? When? What do you think about those results? Are these results important for the region where your organization works? Why? NEEDS IN COUNTRY TO CONDUCT OM SURVEYS What are the skills needed in your organization to conduct this kind of data collection (statistics, research, and public health expertise)? At what levels could the results be used and for what purpose? What are the characteristics of the best-suited organization for this type of data collection and analysis? What are the main limitations to use the OM approach in your country? Cost, time constrains, technical expertise (human resources), decisionmakers’ support? What are your recommendations for expanding the use of the OM approach? (Training, tools, policy?) LQAS OM Based on your experience, what are the main benefits and challenges of using the OM approach? What modifications to the approach are needed to make it easier to implement? What modifications to the approach are needed to make it less costly? Assessment of GH Outcome Monitoring Pilot 41 APPENDIX C: PERSONS AND ORGANIZATIONS CONTACTED Persons and Organizations Contacted in Madagascar Name Position Organization Phone Wendy Benazerga Team Leader HPN USAID 020 22 539 20 Volcan Cakir Deputy Director SanteNet 020 22 289 67 Philippe Lemay Project Director SanteNet 033 02 00145- 22 289 77 Rahelimalala Robertine Coordinator PENSER 032 07 202 81 – 22 644 74 Ratovonanahary Raseliarison Executive Director PENSER 032 04268 87 Douglas E. Call Director PSI 020 22 629 84-032 0745230 Iarimalanto Rabary Research M&E Director PSI 032 0745230 Andry Rabemanantsoa RH Quantitative Research Coordinator PSI 020 22 62984 Christopher Bessey Country Representative CRS 032 07 156 80- 020 22 206 66 Patrick Rakotomahefa M&E Specialist CRS 020 22 206 66 Johan Razafiarison Deputy Operation Manager CRS 020 22 206 66 Jennifer Loucks Deputy Director CARE 032 07 300 72- 020 2233885 Rasamihajamanana Eugénie Director of MCHC MOH 032 04 726 95 Olga Rabemanantsoa Deputy Director Family Planning MOH 032 04 726 95 Razefindravony Bakolisoa Deputy Director of FP in Charge of Dissemination MOH 032 04 726 95 Razetriari Vony Deputy Director of FP in Charge of Supply MOH 032 04 726 95 Peter Delhove Country Director ADRA 033 2384117- 020 2252253 Mamiseheno Rasolofonirina M&E Coordinator ADRA 020 2252253 Josea Ratsirarson Project Director MCDI 033 12 834 85-0202235806 Jacob Ramifehiarivo Training and Partnership Coordinator HIP/AED 020 2425151 Clement Randriatelomanana M&E Coordinator HIP/AED 020 2425151 Razafimandimby Adrianmandrato General Manager Voahary Salama 020 24 202 11 Randrianavoson Rado M&E Specialist Voahary Salama 020 24 202 11 42 Assessment of GH Outcome Monitoring Pilot Persons and Organizations Contacted in Guatemala Name Position Organization Phone Julie Richards Team Leader HPN USAID Baudilio Lopez FSN, HPN USAID (502) 2422-4000 Sergio Penados FSN, HPN USAID (502) 2422-4000 Isabel Stout Project Officer USAID (502) 2422-4000 Fidel Arevalo FSN, HPN USAID (502) 2422-4000 Carlos Rene Bauer Robles Director Ejecutivo APROFAM (502) 2230-5488 Selvin Fuentes Planificacion y Monitoreo APROFAM (502) 2230-5488 Tobin Nelson Gerente de Desarrollo de Cooperación Asociación SHARE (502) 7828-2626 Lheslye Perez M&E Coordinator Asociación SHARE (502) 7828-2626 Rodrigo Bustamante Director Calidad en Salud (502) 2384-0726 Ana Maria Rodas M&E Specialist Calidad en Salud (502) 2384-0726 Jorge Matute Director CIENSA (502) 2472-8501 Laura Leon Investigadora Asociada CIENSA (502) 2472-8501 Hector Gomero Vicepresidente CIENSA (502) 2472-8501 Olga Torres Presidenta CIENSA (502) 2472-8501 Jorge A. Solórzano Director PNUD/GUA/05/027 (502) 2251-9697 Wermer Figueroa Asesor Monitoreo y Evaluación PNUD/GUA/05/027 (502) 2251-9697 Monica M. Rodríguez Gerente Diseno, M&E CRS (502) 2362-2173 Rodrigo Arias Gerente Programa Seguridad Alimentaria Save the Children (502) 2369 6767 Claudia Nieves Lider de Salud y Nutrición Save the Children (502) 2369 6767 Pamela Garrido Coordinadora M&E Save the Children (502) 2369 6767 Marcos Catsam Sub-Director de Programas Save the Children (502) 2369 6767 Alejandro Silva Coordinador Nacional Programa Nacional de Salud Reproductiva Ministry of Health (502) 2472-3407 Julio Garcia Colindres Epidemiólogo Programa Nacional de Salud Reproductiva Ministry of Health (502) 2472-3407 Edwin Montufar Asistente Técnico Programa Nacional de Salud Reproductiva y SIAS Ministry of Health (502) 2472-3407 Assessment of GH Outcome Monitoring Pilot 43 Persons Contacted in Washington, D.C. Name Position Organization Phone Ellen Starbird GH/PRH Deputy Director USAID (202) 712 0847 Al Bartlett Senior Child Health Advisor USAID (202) 712 0991 Mihira Karra GH/PRH USAID (202) 712 5934 Mary Ellen Stanton GH USAID (202) 712 4208 Eunyong Chung FANTA Project CTO USAID (202) 712 4786 Anne Swindale Director FANTA Project (202) 884 8926 Gilles Bergeron Deputy Director FANTA Project (202) 884 8941 Megan Deitchler Senior MCHN M&E Specialist FANTA Project (202) 884 8370 Alison Tumilowicz MCHN M&E Specialist FANTA Project Lisa Maniscalo MEASURE Elizabeth Berard Program Manager GH Tech (202) 884 8722 Barry Silverman Deputy Director GH Tech (202) 884 8722 44 Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot 45 APPENDIX D: REFERENCE MATERIALS/DOCUMENTS CONSULTED FANTA Project Files and Documents Enumerator (CIENSA) training workshop in Guatemala Enumerator (PENSER) training workshop in Madagascar FANTA Guatemala Trip Reports (March 19–30, May 14–25, July 1–6, 2007) FANTA Madagascar Trip Report (April 20–May 12, 2007) GH Outcome Monitoring Indicators Global Health Outcome Monitoring (OM) Overview Indicators Reported to USAID Missions in Guatemala and Madagascar Issues to Consider in Selecting Indicators Layers for Title II List of LAYERS Indicators Madagascar LQAS Simulations Outcome Monitoring Pilot Cost (Madagascar and Guatemala) Outcome Monitoring Survey Sampling Methodology Questionnaires and software for Guatemala Questionnaires and software for Madagascar Results from 2006 Madagascar Survey SPSS syntax and data Sub-agreement with PENSER (Madagascar) and CIENSA (Guatemala) USAID Expert Meeting, September 26, 2006 – Annual Monitoring of Health Outcome Indicators: Assessment of Alternative Data Collection Methodologies 46 Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot 47 APPENDIX E: LIST OF OM INDICATORS POPULATION COMPONENT INDICATORS Table 1: GH-Defined Outcome Monitoring Indicators Malaria17 1. % of households with a child(ren) under age 5 with at least one ITN 2. % of children under 5 years of age in malaria-risk areas reported as sleeping under ITN the previous night 3. % of women who received 2 or more doses of SP for IPT for malaria during their last pregnancy 4. % of children under 5 years of age with fever in last 2 weeks who received antimalarial treatment within 24 hours from onset of fever Maternal and Child Health (MCH) 5. % of women who gave birth who had a PP visit within 3 days 6. % of newborns receiving essential newborn care 7. % of women seen at ANC at least 4 times during their last pregnancy with a live birth 8. % of births attended by a doctor, nurse, or trained midwife (excludes traditional birth attendants) 9. % of children between 12 and 23 months of age who received their third dose of DPT by age 12 months 10. % of children ages 12-23 months receiving a vitamin A supplement during the last six months before the survey 11. % of children under age 5 who are more than 2 SD below the median weight for that age 12. % of infants aged less than 6 months who were exclusively breastfed in the past 24 hours 13. % of children ages 0-59 months with diarrhea in the past 2 weeks who were treated with ORS 14. % of children ages 0-59 months with chest-related cough and fast and/or difficult breathing in the last 2 weeks who were taken to an appropriate health provider 15. % of children ages 12-23 months fed according to a minimum standard of infant and young child feeding practices. Family Planning/Reproductive Health (FP/RH) 16. % of women of reproductive age and sexually active using, or whose partner is using, a modern method of contraception 17. % of need satisfied by modern method of family planning 18. % of women of reproductive age stating their desire to space birth intervals 36 months or longer, or to limit births Table 2: Mission-Defined Outcome Monitoring Indicators Malaria 1. % of women who know how malaria is transmitted 2. % of women who know that pregnant women and children under 5 are at greatest risk if they have malaria 3. % of women who know at least 3 effective ways of preventing malaria 4. % of women who know how to recognize danger signs associated with malaria 5. % of women who know the proper treatment to give to a child with malaria 6. % of women who state knowing where to obtain a long-lasting treated net (LLTN) nearby 7. % of women who state that the price of the locally promoted LLTN is affordable 8. % of women who state knowing the locally promoted malaria prophylaxis 9. % of women who state knowing where to obtain the locally promoted malaria prophylaxis nearby 10. % of women who state that the price of the locally promoted malaria prophylaxis is affordable Maternal and Child Health (MCH)18 11. % of women who received 2 TT shots (or equivalent) during their pregnancy 12. % of women who gave colostrum to their child immediately after birth 13. % of women for whom a clean delivery kit or equivalent was used at the birth of their child 17 Note that the USAID Mission in Guatemala does not support a malaria intervention, due to low incidence of this problem in the Western Highlands where the USAID program is concentrated. Thus, malaria-related indicators were not collected in Guatemala. 18 In all cases, the child mentioned in the indicator wording refers to the index child for this interview. It is not necessarily the youngest child but is never older than 12 months. 48 Assessment of GH Outcome Monitoring Pilot 14. % of women who know the correct dose and frequency for taking iron folate during their pregnancy 15. % of women who know they should have at least 3 prenatal visits with a qualified provider during their pregnancy 16. % of women who can cite at least 3 ways in which they can protect their health and the health of their baby during pregnancy 17. % of women who state they took iron folate once a day during their entire pregnancy 18. % of women who state they took vitamin A less than 40 days after delivery of their child 19. % of women who state knowing where to obtain vitamin A nearby 20. % of women who state knowing where to obtain iron folate nearby 21. % of women who can state at least 2 sources of food that are rich in vitamin A Family Planning (FP) 22. % of women who state knowing at least 1 modern family planning method 23. % of women who state knowing about the contraceptive pill 24. % of women who state knowing where to obtain contraceptive pills nearby 25. % of women who state that the price of contraceptive pills is affordable Reproductive Health and HIV (RH/HIV) 26. % of women who state knowing about HIV 27. % of women who can describe HIV correctly 28. % of women who know how HIV is transmitted 29. % of women who know how to avoid being infected by HIV 30. % of women who state knowing about other STIs OUTCOME MONITORING SURVEY HEALTH CENTER COMPONENT19 Table 3: Indicators Reported to USAID Missions in Guatemala and Madagascar on the Quality of Services offered in Health Centers Supported By USG Resources WELCOMING SERVICES AND USE OF INFORMATION20 1. % of HC that offer IEC sessions to clients while they wait 2. % of HC that visibly display information on the health themes they cover 3. % of HC that take effective steps to reduce client wait time 4. % of HC that visibly display opening and consultation hours 5. % of HC that periodically evaluate client satisfaction 6. % of HC that analyze the information on services rendered to make decisions at the local level HUMAN RESOURCES AND TECHNICAL PLATFORM 7. % of HC that maintain minimum standards of personal comfort for clients in the waiting area 8. % of HC that have an adequate toilet 9. % of HC that have an adequate examination room 10. % of HC that have the adequate equipment to offer expected services 11. % of HC that have the necessary drugs and medications to offer expected services 12. % of HC that display a description of tasks for service providers HYGIENE AND PREVENTION OF INFECTIONS 13. % of HC that have a source of clean water 14. % of HC where the various public spaces were observed to be clean 15. % of HC with an adequate system for the temporary storage of medical waste and soiled cutting/blade instruments 16. % of HC that that visibly display instructions on hygiene, decontamination, and sterilization 17. % of HC that follow national norms for the storage and management of drugs and consumables 18. % of HC that use an adequate process for decontamination and sterilization of instruments 19. % of HC whose cleaning staff wear adequate protective equipment 19 The type of facilities where this component of the survey is carried out corresponds to Health Centers I and II in Guatemala (Centros de Salud) and to Basic Health Centers (Centres de Sante de Base) in Madagscar. 20 Several indicators must combine a number of elements before they are considered satisfactory. See questionnaires for details. Assessment of GH Outcome Monitoring Pilot 49 FAMILY PLANNING 20. % of HC that offer the minimum packet of activities for the provision of FP 21. % of HC that use the standard filing system for FP records 22. % of HC that collect information on the provision of FP services according to national norms 23. % of HC that have at least one qualified staff trained in FP counseling 24. % of HC that have an updated copy of the national directives, norms, and standards for the provision of FP services 25. % of service providers that cited correctly the key steps to follow at the time of first FP consultation 26. % of FP service providers stating that they were never offered money, gifts, or other incentives in exchange for their support in promoting a specific FP method or product 27. % of FP service providers stating that they were never offered money, gifts, or other incentives in exchange for their support in promoting FP in general 28. % of FP service providers stating that they never denied service to someone who refused to use FP STI 29. % of HC that offer the minimum packet in the provision of STI/HIV services 30. % of HC that have qualified human resources for the provision of STI services 31. % of HC in which case management algorithms are available and service providers can access them at any time 32. % of HC that correctly stated the key steps to accomplish at the time of a first STI consultation (syndromic approach PRE/POSTNATAL SERVICES 33. % of HC that that apply the national norms for the provision of pre/postnatal services 34. % of HC that that fill and store patient records adequately 35. % of HC that have at least one person with the qualified staff to deliver pre/postnatal services 36. % of HC that keep daily notation of prenatal consultations 37. % of HC that prepare a consolidated monthly report of TT vaccinations 38. % of HC that offered at least one education session during the last week on one theme related to pre/postnatal care IMCI 39. % of HC that apply national norms for the provision of IMCI services 40. % of HC where service providers fill the registry and monthly reports completely and clearly 41. % of HC that have at least one person trained in IMCI 42. % of HC that have an updated copy of the national directives, norms, and standards for the provision of IMCI services 43. % of HC where the vaccination file drawer is present, in order, and easy to consult 44. % of HC that have a functioning cold chain 45. % of HC that have DPT3 in stock 46. % of HC that offered at least one IEC session in the last week on one of the IMCI themes 50 Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot 51 APPENDIX F: GLOBAL HEALTH OUTCOME MONITORING (OM) SURVEYS What is an OM Survey? USAID’s Bureau for Global Health (GH) is piloting Outcome Monitoring (OM) Surveys, a methodology for data collection by USAID Missions on a set of indicators that allows the USG to monitor the key health activities it supports and to facilitate the management of those activities in country. Health sectors covered by the pilot OM Surveys include malaria; maternal and child health and nutrition; and family planning. OM Surveys have been piloted in Madagascar and Guatemala. GH has designed an external assessment of the pilots that is examining the feasibility of conducting annual or biannual OM surveys; the usefulness of the data for program management and reporting; local capacity issues; and USAID Mission and USAID/Washington bureau interest in such outcome data collection. Why conduct an OM Survey? Every five years, most Missions implement a nationally representative Demographic and Health Survey (DHS) to collect population-level data on health and family planning knowledge, attitudes, and outcomes. However, the new USG Foreign Assistance Framework (FAF) requires annual reporting on indicators that can be attributed to USG-supported interventions, which has largely resulted in output-level reporting. The OM pilot was developed as a rapid and low-cost survey method to be carried out on a yearly basis to (i) complement the output reporting under the FAF with outcome indicators and data attributable to USG investments; and (ii) inform and improve ongoing program implementation. What indicators do OM Surveys collect? The pilot OM Surveys include both population-based and facility-based components. The population-based component collects information on the target population’s knowledge, practices, and access to health services and products, while the facility-based component collects data on the provision of facility-based services and products offered by USG implementing partners in country. The pilot OM Surveys collect information from both components and include two sets of indicators. The first is a set of outcome indicators selected by GH team leaders in each sector (see table below). The second set is selected by in-country Mission staff in consultation with implementing partners to cover the additional data they need to manage their activities. 52 Assessment of GH Outcome Monitoring Pilot GH Outcome Monitoring Indicators Malaria 1. % of households with a child(ren) under age 5 with at least one ITN 2. % of children under 5 years of age in malaria-risk areas reported as sleeping under ITN the previous night 3. % of women who received 2 or more doses of SP for IPT for malaria during their last pregnancy 4. % of children under 5 years of age with fever in last 2 weeks who received antimalarial treatment within 24 hours from onset of fever Maternal and Child Health (MCH) 5. % of women who gave birth who had a PP visit within 3 days 6. % of newborns receiving essential newborn care 7. % of women seen at ANC at least 4 times during their last pregnancy with a live birth 8. % of births attended by a doctor, nurse, or trained midwife (excludes traditional birth attendants) 9. % of children between 12 and 23 months of age who received their third dose of DPT by age 12 months 10. % of children ages 12-23 months receiving a vitamin A supplement during the last six months before the survey 11. % of children under age 5 who are more than 2 SD below the median weight for that age 12. % of infants aged less than 6 months who were exclusively breastfed in the past 24 hours 13. % of children ages 0-59 months with diarrhea in the past 2 weeks who were treated with ORS 14. % of children ages 0-59 months with chest-related cough and fast and/or difficult breathing in the last 2 weeks who were taken to an appropriate health provider 15. % of children ages 12-23 months fed according to a minimum standard of infant and young child feeding practices. Family Planning/Reproductive Health (FP/RH) 16. % of women of reproductive age and sexually active using, or whose partner is using, a modern method of contraception 17. % of need satisfied by modern method of family planning 18. % of women of reproductive age stating their desire to space birth intervals 36 months or longer, or to limit births How do OM Surveys work? OM Surveys are based on an adaptation of lot quality assurance sampling (LQAS) principles. This allows a considerable reduction in the sample size needed to derive statistically reliable data. The entire USG target area is subdivided into Program Management Areas (PMAs), corresponding to a geographical subregion or a partner group. Each PMA constitutes its own LQA sample, which allows assessment of each PMA against a predetermined threshold or performance benchmark. This enables the Mission to judge how well each PMA is doing in the provision of services and to compare performance among them. Further, the aggregation of samples across PMAs yields a total sample size that is usually large enough to calculate parameters (such as the mean) of a particular indicator (e.g., immunization rate or contraceptive prevalence rate) for the entire USAID-targeted area. All data collection is carried out using handheld computers known as personal digital assistants (PDAs), a user-friendly approach that improves accuracy in the field while eliminating post-collection data processing time and costs. Once data collection is complete, a set of tabulations is generated in a simple process that allows for straightforward compilation of the final report. What is involved at the country level? The OM process includes a series of workshops to determine the sample groups and the indicators to be collected and to define a number of PMAs for the survey. In pilot OM surveys, the GH-managed Food and Nutrition Technical Assistance (FANTA) Project has provided technical assistance to Missions throughout this stage. Once the sample groups, indicators, and Assessment of GH Outcome Monitoring Pilot 53 PMAs have been determined, the OM Survey is conducted by an independent third party, recruited locally through a competitive process (RFP). To build local capacity, FANTA trains the local firm to conduct the survey so that OM can be replicated with minimum support from Washington in subsequent years. Equipment such as PDAs and computers need to be purchased either by the USAID Mission or by the contractor. How long do OM Surveys take? This process, including the issuance of an RFP and selection of a firm, can take from eight to 10 months in the first year to allow for the development of prototypes and setting of parameters. Once an OM system is in place, however, the process can be carried out in six to eight months. What do OM Surveys cost? Based on pilot tests of the methodology in Madagascar and Guatemala, the OM process, (including the workshops with the Mission and implementing partners, selection of local firms, training of enumerators and fieldwork, through production of the final report) is estimated to cost about US$150,000 in the first year, including equipment costs. The cost may vary in relation to the size of the bilateral program and the resulting increased equipment needs and will be lower in subsequent years. 54 Assessment of GH Outcome Monitoring Pilot Assessment of GH Outcome Monitoring Pilot 55 APPENDIX G: GUATEMALA AND MADAGASCAR–KEY INDICATORS COMPARISON OF CI Guatemala Indicator Estimate 95% CI Upper Limit CI Spread N Lower Limit Antenatal Care Child dried/wiped and wrapped in warm cloth or blanket immediately and put to the breast 0.36 0.29 0.43 0.14 105 Delivered with assistance of health professional 0.43 0.31 0.56 0.25 105 Respondent and her baby checked by a health professional within 3 d of delivery either at facility or home 0.31 0.05 0.21 0.16 105 Had at least 4 prenatal consultations with health professional 0.42 0.30 0.53 0.24 105 Family Planning Want to limit births and using modern family planning method 0.49 0.37 0.62 0.25 98 Child Nutrition WAZ less than -2 0.26 0.04 0.18 0.14 105 Breastfed and non-breastfed child fed minimum of appropriate feeding practices 0.65 0.57 0.72 0.15 105 Are currently exclusively breastfeeding the child 0.41 0.31 0.52 0.21 105 Vitamin A Received a dose of vitamin A within the last 6 m according to the card 0.67 0.55 0.78 0.24 84 Received a dose of vitamin A in the last 6 m as reported by mother 0.67 0.55 0.79 0.24 64 Received a dose of vitamin A in the last 6 m marked on the card or reported 0.62 0.51 0.73 0.22 105 Immunizations Immunization card shows DPT 3 vaccinations 0.86 0.79 0.92 0.13 95 Received at least 3 DPT vaccinations as reported by mother -0.29 15 Received at least 3 DPT vaccinations marked on card or reported 0.81 0.73 0.90 0.17 105 Child Illness Given the appropriate oral rehydration liquid 0.71 0.62 0.80 0.17 105 Taken to appropriate health provider when sick with cough and fast and/or difficult breathing 0.10 0.02 0.18 0.16 69 56 Assessment of GH Outcome Monitoring Pilot Madagascar Indicator Estimate 95% CI Upper Limit CI spread N Lower Limit Antenatal Care Child dried/wiped and wrapped in warm cloth or blanket immediately and put to the breast 0.45 0.32 0.59 0.27 84 Delivered with assistance of health professional 0.39 0.23 0.55 0.32 84 Respondent and her baby checked by a health professional within 3 d of delivery either at facility or home 0.34 0.21 0.47 0.26 84 Had at least 4 prenatal consultations with health professional 0.45 0.33 0.57 0.25 84 Family Planning Want to limit births and using modern family planning method 0.51 0.40 0.62 0.22 84 Child Nutrition WAZ less than -2 0.38 0.29 0.47 0.19 84 Breastfed and non-breastfed child fed minimum of appropriate feeding practices 0.44 0.32 0.57 0.26 84 Are currently exclusively breastfeeding the child 0.63 0.50 0.76 0.26 84 Vitamin A Received a dose of vitamin A within the last 6 m according to the card 0.88 0.80 0.97 0.16 46 Received a dose of vitamin A in the last 6 m as reported by mother 0.85 0.73 0.96 0.22 43 Received a dose of vitamin A in the last 6 m marked on the card or reported 0.88 0.79 0.96 0.17 84 Immunizations Immunization card shows DPT 3 vaccinations 0.94 0.87 1.01 0.14 60 Received at least 3 DPT vaccinations as reported by mother 0.67 0.41 0.93 0.53 25 Received at least 3 DPT vaccinations marked on card or reported 0.87 0.77 0.97 0.20 84 Child Illness Given the appropriate oral rehydration liquid 0.34 0.23 0.45 0.22 84 Taken to appropriate health provider when sick with cough and fast and/or difficult breathing 0.33 0.17 0.49 0.32 51 Malaria Have at least 1 qualified bednet in home 0.64 0.48 0.80 0.31 84 Stated reference child slept under qualified bednet previous night 0.62 0.47 0.78 0.31 84 Received at least 2 doses of SP/Fansidar during last pregnancy 0.35 0.22 0.49 0.28 84 Received antimalarial treatment within 24 h from onset of fever 0.11 0.03 0.19 0.16 84 Assessment of GH Outcome Monitoring Pilot 57 Comparison Guatemala Madagascar Indicator 95% CI 95% CI Estimate Lower Limit Upper Limit CI Spread Estimate Lower Limit Upper Limit CI spread Diff. in CI Antenatal Care Child dried/wiped and wrapped in warm cloth or blanket immediately and put to the breast 0.36 0.29 0.43 0.14 0.45 0.32 0.59 0.27 0.13 Delivered with assistance of health professional 0.43 0.31 0.56 0.25 0.39 0.23 0.55 0.32 0.07 Respondent and her baby checked by a health professional within 3 days of delivery either at facility or home 0.31 0.05 0.21 0.16 0.34 0.21 0.47 0.26 0.10 Had at least 4 prenatal consultations with health professional 0.42 0.30 0.53 0.24 0.45 0.33 0.57 0.25 0.01 Family Planning Want to limit births and using modern family planning method 0.50 0.37 0.62 0.25 0.51 0.40 0.62 0.22 - 0.03 Child Nutrition WAZ less than -2 0.26 0.04 0.18 0.14 0.38 0.29 0.47 0.19 0.05 Breastfed and non-breastfed child fed minimum of appropriate feeding practices 0.65 0.57 0.72 0.15 0.44 0.32 0.57 0.26 0.11 Are currently exclusively breastfeeding the child 0.41 0.31 0.52 0.21 0.63 0.50 0.76 0.26 0.05 Vitamin A Received a dose of vitamin A within the last 6 months according to the card 0.67 0.55 0.79 0.24 0.88 0.80 0.97 0.16 - 0.08 Received a dose of vitamin A in the last 6 months as reported by mother 0.67 0.55 0.79 0.24 0.85 0.73 0.96 0.22 - 0.02 Received a dose of vitamin A in the last 6 months marked on the card or reported 0.62 0.51 0.73 0.22 0.88 0.79 0.96 0.17 - 0.05 Immunizations Immunization card shows DPT 3 vaccinations 0.86 0.80 0.92 0.13 0.94 0.87 10.1 0.14 0.01 Received at least 3 DPT vaccinations as reported by mother -0.29 0.67 0.41 0.93 0.53 Received at least 3 DPT vaccinations marked on card or reported 0.81 0.73 0.90 0.17 0.86 0.77 0.96 0.20 0.03 58 Assessment of GH Outcome Monitoring Pilot Comparison Guatemala Madagascar Indicator 95% CI 95% CI Estimate Lower Limit Upper Limit CI Spread Estimate Lower Limit Upper Limit CI spread Diff. in CI Child Illness Given the appropriate oral rehydration liquid 0.71 0.62 0.80 0.17 0.34 0.23 0.45 0.22 0.05 Taken to appropriate health provider when sick with cough and fast and/or difficult breathing 0.10 0.02 0.18 0.16 0.33 0.17 0.49 0.32 0.15 Malaria (reported for Madagascar) Have at least 1 qualified bednet in home 0.64 0.48 0.80 0.31 Stated reference child slept under qualified bednet previous night 0.62 0.47 0.78 0.31 Received at least 2 doses of SP/Fansidar during last pregnancy 0.35 0.22 0.49 0.28 Received antimalarial treatment within 24 hours from onset of fever 0.11 0.03 0.18 0.16 Assessment of GH Outcome Monitoring Pilot 59 APPENDIX H: REFERENCES Abbas Bhuiya, S.M.A.H., Nikhil Roy, & P. Kim Streatfield. 2007. “Performance of the Lot Quality Assurance Sampling Method Compared to Surveillance for Identifying Inadequately-performing Areas in Matlab, Bangladesh.” Journal of Health Population and Nutrition 25(1): 37-46. Child Survival and Health Program data base maintained by Macro International, Child Survival Technical Support Plus (CSTS+) Crespin, Xavier. 2005. “Knowledge, Practice, Coverage, Baseline Survey Report.” Helen Keller International/Niger. Grundmann, Christophe. 2002. “The Costs of Using LQAS for Project Management, Monitoring and Evaluation.” NGO Networks for Health Project. Olives, Casey. 2006. Madagascar LQAS Simulations. Department of Biostatistics. Harvard School of Public Health. August 24, 2006. Robertson, S.E. & J.J.Valadez. 2006. “Global review of health care surveys using lot quality assurance sampling (LQAS), 1984–2004.” Social Science & Medicine 63: 1648– 1660. USAID. 2003. Regional Strategy for Central America and Mexico FY 2003-2008. Volume 2: Annex E: Guatemala Country Plan. Valadez, J. J. 1991. Assessing child survival programs in developing countries: Testing lot quality assurance sampling. Boston: Harvard School of Public Health. Valadez, J. J., W.Weiss, C. Leburg, & R. Davis. 2003. Assessing Community Health Programs: Using LQAS for Baseline Surveys and Regular Monitoring. St Albans, England: Teaching-aids at Low Cost. 60 Assessment of GH Outcome Monitoring Pilot For more information, please visit http://www.ghtechproject.com/resources/ Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com