EXTERNAL EVALUATION OF THE MEASURE PHASE III DEMOGRAPHIC AND HEALTH SURVEYS (DHS) PROJECT FINAL REPORT MAY 24, 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Richard Cornelius, Ann Larson and Andrew Kantner through Management Systems International. EXTERNAL EVALUATION OF THE MEASURE PHASE III DEMOGRAPHIC AND HEALTH SURVEYS (DHS) PROJECT FINAL REPORT Contracted with PHI under Cooperative Agreement # GPO-A-00-06-00005-00 Global Health Fellows Program Technical and administrative support was provided to the Team through MSI staff Carrie C. Foti, MPH and Maria Zapata. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EXTERNAL EVALUATION OF MEASURE DHS i CONTENTS ACRONYMS AND ABBREVIATIONS ...................................................................................... ii I. EXECUTIVE SUMMARY ....................................................................................................... iv II. BODY OF THE REPORT ....................................................................................................... 1 1. Background ..................................................................................................................... 1 2. Purpose of the Evaluation ............................................................................................... 2 3. Team Composition .......................................................................................................... 3 4. Methodology ................................................................................................................... 3 A. Stakeholder Interviews ...................................................................................... 3 B. Country Case Studies ........................................................................................ 4 C. E-Surveys of USAID Missions and Presidential Malaria Initiative (PMI) Advisor....................................................................................................................4 D. Review of MEASURE DHS Reports, Documentation, and Online Materials.......................... ......................................................................... 5 E. Analysis of Results............. ............................................................................... 5 F. Limitations of the Methods ................................................................................ 5 5. Main Findings ................................................................................................................. 6 A. Meeting Stakeholders’ Needs ........................................................................... 6 B. Effectiveness of Technical Assistance ............................................................ 14 C. Task 2: Effectiveness of Capacity Building Activities ................................... 17 6. Conclusions ................................................................................................................... 23 A. Meeting Stakeholders’ Needs ......................................................................... 23 B. Effectiveness of Technical Assistance ............................................................ 24 C. Task 2: Effectiveness of Capacity Building Activities ................................... 25 7. Recommendations ......................................................................................................... 25 A. Summary Key Recommendations ................................................................... 25 B. Detailed Recommendations/Discussion by Task ............................................ 27 APPENDIXES .............................................................................................................................. 32 A. Statement of Work ....................................................................................................... 32 B. List of USAID Mission Respondents Reached via E-survey....................................... 44 C. PMI Contacts Reached via E-survey............................................................................ 46 D. List of Documents Reviewed....................................................................................... 47 E. List of Key Informants Interviewed, by Organization ................................................. 55 F. Data Collection Instruments Developed....................................................................... 61 G. Country Selection Criteria ........................................................................................... 73 H. Country Case Descriptions........................................................................................... 74 I. Tabulations from the E-survey ...................................................................................... 85 J. PowerPoint Presentation on the Results of the Evaluation............................................ 97 EXTERNAL EVALUATION OF MEASURE DHS ii ACRONYMS AND ABBREVIATIONS ACT Artemisin-based combination treatment AIS AIDS Indicator Survey ARV Anti-Retroviral AusAid Australian Agency for International Development BDHS Bangladesh Demographic Health Survey CDC Centers for Disease Control and Prevention CDHS Cambodia Demographic Health Survey CHSU Community Health Sciences Unit – Ministry of Health Malawi CSPro Census and Survey Processing System CTO Cognizant Technical Officer DFID Department for International Development (United Kingdom) DHS Demographic Health Survey DSS Demographic Surveillance System E&E Eastern Europe and Eurasia GH Global Health GIS Geographic Information System GTZ Deutsche Gesellschaft für Internationale Zusammenarbeit HMIS Heath Management Information System HQ Headquarters HSSP2 Second Health Sector Support Program ICDDR, b International Centre for Diarrhoeal Disease Research, Bangladesh ICF ICF International IPTP Intermittent anti-malarial drug treatment for pregnant women ITN Insecticide-Treated Net IUSSP International Union for the Scientific Study of Population JICA Japan International Cooperation Agency LIA Local Implementing Agency M&E Monitoring and Evaluation MASEDA Malawi Social and Economic Database MDHS Malawi Demographic Health Survey MICS Multiple Indicator Cluster Survey MIS Malaria Indicator Survey MOH Ministry of Health EXTERNAL EVALUATION OF MEASURE DHS iii MOHFW Ministry of Health and Family Welfare (Bangladesh) MOHSW Ministry of Health and Social Welfare (Tanzania) NAC National AIDS Commission (Malawi) NBS National Bureau of Statistics (Tanzania) NGO Non-Governmental Organization NIPORT National Institute for Population Research and Training NIS National Institute of Statistics (Cambodia) NSC National Steering Committee (Malawi) NSO National Statistical Office OJT On the Job Training PAA Population Association of America PDA Personal Digital Assistant PEPFAR President’s Emergency Plan for Aids Relief PMI President’s Malaria Initiative PCR Polymerase Chain Reaction SMC Social Marketing Company (Bangladesh) SPA Service Provision Assessment Survey SWAp Sector-Wide Approach TA Technical Assistance TB Tuberculosis TDHS Tanzania Demographic Health Survey TFNC Tanzania Food and Nutrition Center THMIS Tanzania HIV and Malaria Integrated Survey TOT Training of Trainers UNAIDS Joint United Nations Programme on HIV/ AIDS UNFPA United Nations Population Fund UNGASS United Nations General Assembly Special Session on HIV/AIDS UNICEF United Nations Children’s Fund USAID United States Agency for International Development USAID/W United States Agency for International Development Headquarters USG United States Government VCT Voluntary Counseling and Testing WHO World Health Organization EXTERNAL EVALUATION OF MEASURE DHS iv I. EXECUTIVE SUMMARY The Demographic and Health Surveys (DHS) has been USAID’s flagship global health data collection project since 1984. The environment in which DHS surveys are now conducted has changed considerably since the 1980s. The emergence of HIV/AIDS, increased efforts to control malaria and the Millennium Development Goals on maternal and child health are among the new global health priorities for which the DHS provides high quality, internationally comparable, and accessible data for program planning, management, monitoring, and evaluation. Questionnaires have become longer and more complex and include a much wider array of health and nutrition data. Many partnering countries demand larger samples that will provide more stable estimates for decentralized health systems. This evaluation addresses three tasks per the statement of work (See Appendix A, p. 33). The first task is to analyze the costs and benefits of how MEASURE DHS operates within this complex and changing environment with an emphasis on contractor’s success in delivering two aspects of their operation: (a) responding to the varied needs of key stakeholders, and (b) providing technical assistance (TA) in survey content, design, and implementation. The evaluation team identified needs within the manageable interests of the project and TA approaches that might increase contractor’s TA performance (including those that may not be part of the current project approach). The second evaluation task is to assess the feasibility and appropriateness of the capacity building goals laid out in the MEASURE DHS contract in light of the increased complexities of survey work and global demand for high quality and timely information by examining stakeholder (particularly host country stakeholder) needs and advising on approaches that might increase contractor capacity building performance. The third task is to provide recommendations and document what is and is not working with regard to meeting stakeholder needs and conducting TA and capacity building activities. The principal methodologies used by the evaluation team were document review, semi-structured interviews, country case studies, and an e-survey of all countries with a MEASURE DHS survey in the current funding round. The team gathered information from multiple DHS stakeholders groups, namely, the three technical offices of USAID’s Global Health Bureau (which provide core funding for the DHS), USAID’s regional bureaus, USAID overseas missions, other U.S. government agencies, ICF International management and staff, key international organizations, and U.S.-based stakeholders and experts. Field visits to Bangladesh, Cambodia, Malawi, and Tanzania provided an in-depth understanding of how DHS surveys were designed, conducted, analyzed, and dissemination in countries which had at least ten years of involvement in the program. During each country evaluation, team members interviewed officers from the USAID mission, host government representatives from the Ministry of Health, UN agencies, other bilateral donors with a country presence, and major health-related national and international NGOs. Interviews were also conducted with technical experts and managers from the LIA (usually the national statistical office), collaborating laboratories, local researchers, DHS consultants, and participants from previous DHS workshops. In total, 142 people were interviewed during this evaluation. Data from key-informant interviews, documents, country case studies, and e-Survey were managed in Excel spreadsheets. Common themes and divergent views were identified by stakeholder group and triangulated with data from other sources. EXTERNAL EVALUATION OF MEASURE DHS v Summary of Main Findings Task 1.a: Meeting stakeholders’ needs In interviews (See Appendix E, p. 55) and the e-Survey (See Appendixes C & D, pp. 44-46), the review team encouraged stakeholders to describe what features of the DHS they valued and if DHS data met their needs. They were asked how they used the DHS results and if there were aspects about survey content or processes they wanted changed. The evaluation team found that the DHS continues to play a major role in providing essential information on health conditions throughout the world. International organizations, the U.S. government, and stakeholders in participating countries value the data. All stakeholders said they would continue to need the DHS for a long time into the future. Four needs that MEASURE DHS currently satisfy are: • High quality population-based data • Internationally comparable indicators • Country level data for national health planning and monitoring • Data availability through final reports, data sets in a number of formats, and the STATcompiler Meeting these needs comes at a cost. The ceiling of $142 million for the current five-year funding period has nearly been reached in less than four years, despite an additional $41,143,065 million in contributions from other bilateral and multilateral donors and host governments. The cost of the DHS program is not purely economic. Operating surveys in over 90 countries since 1984 have required that MEASURE DHS are very responsive to stakeholders needs while maintaining firm checks on data quality and timeliness. The need to achieve high marks on responsiveness in relation to content and high data quality standards has led to concerns among stakeholders that perhaps a different type of DHS program is necessary. Specifically, the evaluation team heard about four broad needs for population and health surveys that they believed were not being met by MEASURE DHS: • More frequent surveys for timely monitoring of program effects • Controlling the length and sample size of surveys • More policy relevant outputs The first two unmet needs are not manageable by the DHS contractor. Population-wide changes occur relatively slowly and more frequent surveys are unlikely to measure change. Measures of program outputs and short-term impact collected from program information management systems or small surveys would be more programmatically useful and much less expensive. However, there is scope to review the DHS survey schedules with host governments and USAID missions to confirm that the current survey cycle complements the national health planning cycle. At the beginning of MEASURE III DHS, the contract and USAID management teams reduced the core questionnaire. However, neither group has the authority to refuse to include questions that are technically feasible and requested by host-governments or global programs. ICF International is already expanding the types of publications and other output that are done, and there is scope to increase the demand for these outputs within countries. EXTERNAL EVALUATION OF MEASURE DHS vi Lastly, stakeholders identified three broader goals or aspirations for the DHS program. These closely related needs were: • Reducing the complexity and size of the DHS • Investing in health management information management systems and other forms of health data • Host governments, national institutions, and local experts taking greater control of the survey process The three needs concern orienting the DHS to meeting the unique needs of each country for the production and use of population and health data. Some immediate actions could be taken to incorporate a broader country-led approach to ICF International’s provision of technical assistance and capacity building. Significant change will require a change in USAID priorities for future MEASURE DHS. Task 1.b: Technical assistance in survey content, design and implementation DHS’s excellent reputation is attributed to the high quality of the survey process and involvement of international experts through intensive TA from ICF. The majority of DHS are conducted with a “full model” of TA consisting of over 1.5 person years, although the actual amounts vary considerably. The typical full package of TA includes about 10-12 visits to a country in addition to HQ technical support. Surveys with limited TA packages were usually provided with targeted assistance regarding new modules or technologies and occur in countries that have strong survey capacity. One finding of immediate relevance was that the amount and purpose of TA was rarely discussed in developing the work plan. ICF would base their recommendations on an informal assessment of country capacity and experiences from previous surveys. As USAID missions usually pay the TA component of field support, the amount of TA is not formally discussed with members of the Steering or Technical Committees. ICF is currently using a number of modalities to provide TA. The most common form is short-term country visits by headquarter-based ICF staff. ICF has also developed a pool of experienced regional consultants and survey professionals with expertise in biomarkers, sampling, interviewer training, data processing, data analysis, and data use. Often, consultants have worked on DHS in their home countries and continued to develop their skills through other survey work, DHS-sponsored workshops, and opportunities to work at DHS headquarters. Such consultants are less expensive than full time U.S.-based employees and often have relevant cultural and language skills to work within their local regions. National or regional resident advisors are also used to assist local implementing agencies with data collection and management processes. These advisors who have a solid grounding in survey processes are often past employees of the MOH or national statistics office (NSO) and contribute to the quality and timeliness of the results. These consultants are also highly valued. With few exceptions, in-country stakeholders spoke highly of the TA provided by ICF. Consultants and staff members were seen as committed, hard-working, and willing to share their knowledge. Nevertheless, the evaluation team consistently heard from stakeholders that TA prioritized getting the job done rather than assisting national counterparts to do the work required. In three of four case study countries and half of the e-Surveys, respondents indicated that they wanted national institutions and experts to play a greater role in the survey process continuum from design to dissemination. EXTERNAL EVALUATION OF MEASURE DHS vii Task 2: Feasibility and appropriateness of capacity building goals, stakeholder needs and approaches Throughout nearly three decades, many countries have greatly increased their capacity to implement and use household DHS. Countries such as Egypt, Philippines, Peru, Thailand, Colombia, Nepal, and many more conduct their own surveys or use only very limited TA related to new technologies or indicators. Almost all of the technical work for India’s DHS was done by Indian research institutions and companies. The DHS program can take some responsibility for this transformation; although, many other factors are attributed. International, U.S. government, and in-country stakeholders expressed demands for greater national capacity. Commonly expressed views were that countries should have the capacity to collect quality data, manage it effectively, and produce tables and indicators. National capacity to interpret and apply the results was also seen as vital; although, stakeholders believed that external assistance regarding sampling and report writing will continue for some time. ICF produced a Capacity Building Strategy (See Appendix D, p. 47) that included the following target areas: • Sampling • Data processing • Data analysis • Using survey results for program planning The Strategy reflects the challenges in prioritizing capacity building. On the one hand, national capacity to undertake (and even lead) all stages of the survey continuum is an explicit goal; however, the Strategy also indicates that in most cases, it is not feasible due to funding of specific capacity building activities and the complexity of the DHS processes (including both data processing and analysis). ICF International implemented a number of effective capacity building activities in this funding period as well as continuing activities that have worked well in the past. These range from the informal on-the-job training and mentoring to ad hoc workshops (when requested) to strategic activities. The strategic activities show the most promise for long-term success. They include training selected individuals and institutions that will then be responsible for dissemination of results and training others in interpretation and application. Using a combination of Fellowships and centers of excellence has the potential to reduce the demands for external TA and increase national capacity in the production and use of health survey data. Specific findings on capacity building activities are in Table 1. TABLE 1: CAPACITY BUILDING APPROACHES Current Capacity Building Models Definition Views or Evaluator Comments (if new) Costs/Benefits Classroom training/operational In-country didactic sessions and involve a specific DHS curricula and trainer Necessary for the implementation of the survey – interviews, field editing and quality checking, lab testing and recording, and data entry Takes ICF time and resources, can be done by the LIA in many contexts. Is necessary for each DHS cycle. EXTERNAL EVALUATION OF MEASURE DHS viii The evaluation team saw opportunities for incremental improvements in capacity building within the current constraints, principally through the articulation of measurable targets. For instance, the conducting of participatory needs assessments was not evident in country case studies with more significant levels of CB activities even though the process is featured in the capacity building strategy as articulated by ICF to USAID/W. The evaluation team saw great promise in the stated strategy and methods in terms of being relevant and suited to the needs. Future tensions around in-country capacity could be mitigated if baseline needs assessments be conducted with NSOs or other survey implementers and used as a basis for TA planning and other inputs from ICF during and after a survey cycle. Recommendations for future directions (Task 3) and managing the resolution of errors is part of the routine implementation of population-based surveys. Mentoring for advanced skills We also term this “on-the-job” (OJT) training. It includes work on-site with LIA during survey phases This is being done in almost all DHS countries. It is highly valued. Takes additional time and requires additional skills. Like advanced training, it does not address staff turnover and potentially needs to be done each DHS cycle. Classroom training for more advanced skills Didactic sessions and involve a specific DHS curricula and trainer Not widely available in all DHS countries. There is substantial demand for this level of training specific to “black box” topics for future DHS cycles. Does not address human resource issues faced by LIAs and turnover of staff between DHSs. Additionally, advanced skills requires a base knowledge that is not always present. User-friendly Manuals, e￾Learning courses, and website info. Includes DHS Toolkit, a guide to help improve the basic skills to conduct a DHS. Viewed to be very helpful by university and secondary analysis oriented country level stakeholders Needs to be updated/maintained by ICF￾Calverton staff Fellows program Short-term fellowships are offered to selected young researchers through USAID Washington funds. Training takes place at ICF headquarters or another country for fellows from the surrounding region. NA. The sampled responses had few comments or exposure to Fellows program. They have proved to be time￾intensive for fellows and staff who are required for one-on￾one mentoring Other potential models: organizational development Use of institutional development consultants, tools, and methodologies to identify the strengths, weaknesses, and opportunities for LIAs during DHS cycles and beyond. Build consensus and helps identify shared goals, objectives, and activities by country. Could be best applied during study design phase and connect to a broader/global level OD process with ICF, USAID/Washington, and other key partners in the U.S. Consultant LOE, staff time, local travel, coordination at ICF. ICF would have a clear process for capacity building globally and shared goals and milestones by partner entity and country. EXTERNAL EVALUATION OF MEASURE DHS ix Task 3.1: What is working well? The DHS continues to provide high-quality, timely data project-wide. The engagement of stakeholders at international and country-levels through Committees has been successful and should continue. Further, the excitement and energy around DHS implementation is evident and the desire for countries to take on more or in some cases, all of the survey functions based on largely on-the-job capacity building is impressive. It illustrates an appreciation for the work of the ICF team and a desire to better use ICF time and resources. All stakeholders see a continued role for ICF. Technical assistance will continue to be the main strategy for MEASURE III DHS to ensure international comparability, data quality, and timeliness of results. Any significant changes to this model should be made with an understanding of how much TA is provided and for what purposes. Tracking TA by country and survey, as well as globally, will assist in setting targets for its eventual replacement by national institutions. Tasks 3.2 - 3.3: What could be improved or needs changing? The growing size of the DHS along with the demands for smaller geographic estimates and better programmatic data suggests the need for USAID to become much more involved in the support of a broader approach to health data. By only supporting household population surveys, there arises increasing demands to collect more and more indicators about how people interact with the health system and to document their health status through biomarkers. In many cases, these indicators are cost-effective to add to an existing survey, but a population survey may not be the most appropriate method. There is a need to look broadly at the role of household surveys with the aim of complementing surveys with censuses, surveillance systems, and HMIS. This is beyond the scope of MEASURE III DHS to manage but should be considered in future programs. Additionally, clearer technical governance processes should be considered to assist in containing the length of the survey from questions that could be obtained in other ways. This action needs to come from USAID. The contractor’s principal obligation is to be responsive and not to make final decisions; however, the contractor does have a role in producing the evidence upon which technical decisions are made. DHS will remain heavily dependent on TA in the future. A short-term approach to TA concerns would have to address the number and purpose of TA visits and be discussed amongst all stakeholders at the design phase. Local experts could be identified and a job description developed that outlines how they will work with ICF advisors. The evaluators appreciate that some of this information may be commercially sensitive. Future strategies for effective capacity building would be informed through evaluation of factors in some countries that no longer rely on MEASURE DHS TA, including enablers and barriers to producing quality data for their own programming needs and contributing to international collections of comparable, high quality data. Funding mechanisms for survey implementation and sustainable capacity should also be investigated. Capacity building is an ongoing process that does not easily fit with the current cycle of conducting DHS in countries every five years. Future programming should encourage the use of national and regional partners who can source local expertise developed by the contractor when the DHS is active and who continue to have training, dissemination, and analysis roles between surveys. Capacity building should also be measured against global targets and milestones in implementing capacity reached by country partners. Explicit capacity building goals and objectives (individual and institutional) would also be weaved into five-year work plans and strategies. EXTERNAL EVALUATION OF MEASURE DHS 1 II. BODY OF THE REPORT 1. Background The Demographic and Health Surveys Project (also known as MEASURE DHS) was developed to collect, analyze, and disseminate household and facility-based information on demographic and health issues of relevance to planning, policy-making, and the monitoring and evaluation of health program performance (See Appendix D, p. 47). Major objectives of MEASURE DHS include: (1) providing valid information on levels and trends in demographic and health status and (2) improving the ability of partnering countries to undertake high quality household and facility-based surveys to meet basic needs for health information. Further MEASURE III seeks the achievement of six intermediate results: • Result 1: Increased user demand for quality information, methods, and tools for decision-making • Result 2: Increased in-country individual and institutional technical/managerial capacity and resources for the identification of data needs and the collection, analysis, and communication of appropriate information to meet those needs • Result 3: Increased collaboration and coordination in efforts to obtain and communicate health, population, and nutrition data in areas of mutual interest • Result 4: Improved design and implementation of the information gathering process including tools, methodologies, and technical guidance to meet user needs • Result 5: Increased availability of population, health, and nutrition data, analyses, methods, and tools • Result 6: Increased facilitation of use of health, population, and nutrition data. MEASURE DHS is now in the third ten-year phase of its project life. The MEASURE III DHS contract runs from September 30, 2008 to September 29, 2013 with a project budget of US $142,462,784. Since 1984, MEASURE DHS has completed more than 260 surveys in 90 countries, primarily in sub-Saharan Africa, South and Southeastern Asia, and Latin America (MEASURE DHS website). ICF International, headquartered in Calverton, Maryland, is currently implementing the project. There has been a growing tendency in recent years for the DHS surveys to be supported through multiple funding sources. This trend has been encouraged by USAID and largely fueled by increasing costs for surveys as data demands have grown. In the current Phase III of the MEASURE DHS Project, host country governments and other donors have provided US $41,143,065 million (as of September 30, 2011). It is important to note that the environment in which DHS surveys are conducted has changed considerably since the 1980s. The survey questionnaires have become longer and more complex and include a much wider array of health and nutrition data. There is now greater demand in many partnering countries for larger samples that will provide more stable estimates for sub-national areas, especially in countries that have decentralized health systems. The DHS core questionnaire includes detailed information on such topics as reproductive health (family planning, antenatal, delivery, and postnatal care), maternal and child nutrition, client contact with service providers and health facilities, and HIV knowledge and risk behavior. Separate modules are used for EXTERNAL EVALUATION OF MEASURE DHS 2 measuring maternal mortality, domestic violence, female genital cutting, and fistula. The use of surveys for men has also increased the scope of DHS programs in many countries. Special survey questionnaires have been developed to better meet the data needs of individual countries. For example, the AIDS Indicator Survey (AIS) and the Malaria Indicator Survey (MIS) are increasingly used to provide information on seroprevalence, prevention efforts, as well as care and treatment programs. The facility-based Service Provision Assessment Survey (SPA) has also been implemented in several DHS partnering countries (e.g., Bangladesh, Egypt, and Tanzania). 2. Purpose of the Evaluation The purpose of this evaluation is to assess the extent to which the DHS is meeting user needs for demographic and health information, effectiveness of the project’s current TA efforts, and extent of efforts to build host country capacity to undertake and use household and facility-based surveys. Recommendations are provided on how the DHS should be positioned to effectively satisfy future health data needs. The three main tasks identified for this evaluation by USAID are as follows: Task 1: Provide an analysis of the costs/benefits of how the DHS Project operates within this complex and changing environment, with an emphasis on the following: • How successfully the contractor is responding to the varied needs of key stakeholders and which needs are within the manageable interests of the project. • How successfully the contractor is providing TA in survey content, design, and implementation. What TA approaches (including those that may not be part of the current project approach) might increase the contractor's performance in this area? Task 2: Provide an analysis of the feasibility/appropriateness of the capacity building goals laid out in the contract in light of the increased complexity of survey work and global demand for high quality survey data, with an emphasis on the following: • What are the priority survey design/management and data analysis/use capacity building needs of key stakeholders? Are the capacity building strategies and activities undertaken by the project meeting these needs, and in particular, the needs of local implementing agencies (LIAs)? If not, what are the key barriers at different levels (i.e., design of the contract, stakeholder demand, donor funding, and local “ownership”)? • What capacity building approaches (including those that may not be part of the current project approach) might increase the contractor's performance in the capacity building of local institutions and individuals in survey design/management and data analysis/use? Task 3: Provide recommendations for future directions in health data collection and survey design (based on Tasks 1 and 2). EXTERNAL EVALUATION OF MEASURE DHS 3 DHS Other US government USAID Mangement Team DHS USAID leadership ICF International Global and regional stakeholders Figure 1: MEASURE DHS stakeholders interviewed 3. Team Composition The team leader for the evaluation is Richard Cornelius, a demographic and health survey specialist who worked for many years with USAID’s Global Health Bureau. He has extensive experience managing and providing technical oversight for USAID’s data-gathering efforts in population and health. Mr. Cornelius was the Cognizant Technical Officer (CTO) for the DHS Project from its inception in 1984 until 1993. Ann Larson is a technical consultant based in Australia with considerable evaluation expertise in capacity building and organizational development in primary and public health. She worked for many years on various aspects of health survey design, field operations, as well as analysis and research. She spent several years with USAID/Bangladesh as the Mission’s demographic and research management advisor during which she provided technical support for the 1989 Bangladesh Contraceptive Prevalence Survey and the design of the first official Bangladesh Demographic and Health Survey conducted in 1993-1994. Andrew Kantner is the third member of the evaluation team. He has worked extensively with DHS over his career, most notably in Bangladesh, Indonesia, and the Philippines. He helped organize secondary analysis workshops for surveys conducted in these and other countries during the 1990s and has also participated in numerous evaluations for USAID’s Global Health Bureau over the past decade, focusing primarily on reproductive health and HIV/AIDS. 4. Methodology The MEASURE DHS evaluation utilized multiple approaches to gather and analyze quantitative and qualitative information directly relevant to the three evaluation tasks. The principal methodologies used were document review (See Appendix D, p. 47), stakeholder interviews (See Appendixes E & F, pp. 55-60, 61-68), country case studies (See Appendixes G & H, pp. 73-74), and an e-Survey (See Appendixes B, C, I & F, pp. 44-47, pp. 68-72, pp. 85-96) of all countries with a MEASURE DHS survey in the current funding round. A. Stakeholder Interviews The evaluation team received a briefing from the MEASURE DHS management team early in the project to describe the background of the Scope of Work and identify key international stakeholders that use DHS data or provide important resources or technical input into survey design and implementation. Figure 1 displays the five stakeholder groups whose views the evaluation team garnered primarily through face-to-face, semi-structured interviews in the Washington D.C. area. A handful of interviews were conducted by telephone. A complete list of people interviewed is in Appendix E, pp. 55-60 of this Report. The interviews opened with questions to determine each stakeholder’s involvement with the EXTERNAL EVALUATION OF MEASURE DHS 4 DHS program and their role, which ranged from using data to set global health policy agenda to implementing surveys. Following a general framing question, all stakeholders were asked a range of open-ended and probing questions (See Appendix F, pp. 61-68) related to the Scope of Work based on specific question-lines developed prior to the interviews for each stakeholder group. Each interview covered the following broad issues: • Use of DHS survey results and other major sources of data • The adequacy of involving stakeholders in DHS planning, implementation, and use • Effectiveness of the TA model(s) used in MEASURE DHS • Extent of capacity building that occurred through MEASURE DHS and relative importance of further capacity building as a project objective Interviews took an average of 30-45 minutes. Notes taken during the interviews were organized by thematic area into an Excel file and later used for analysis. B. Country Case Studies Four countries (Bangladesh, Cambodia, Malawi, and Tanzania) were visited by one or more evaluation team members. These countries were selected by USAID because each had had at least three DHSs including one during phase three. Case study methodology does not require that cases be representative of the total population of countries with a DHS. The analytical power of case studies is that they give insights into “how” and “why.” They provide rich descriptions of the factors that determine whether the DHS meet stakeholder needs by examining how survey content was negotiated and whether levels of TA were or were not appropriate. The case studies also shed light on how (if at all) the DHS exercise built host country capacity in survey design, collection, and data use and analysis. Opportunities for capacity building within an environment of multiple host government and donor activities were also explored in the interviews. In each case study, country interviews were held with senior personnel involved in DHS design and implementation or individuals who played a key role in funding the survey or using the data. These stakeholders consisted of representatives of the government including the Ministry of Health, relevant line agencies, the national statistical office, and donors with DHS involvement such as USAID, UNICEF, UNFPA, World Bank, DFID, and AusAID. In some countries, NGOs and local researchers were also interviewed as major users or enablers. Particular effort was made to speak with persons who participated in training or other DHS capacity building exercises. A single-question line was used for every in-country interview (See Appendix F, p. 55) but adjusted as appropriate to accommodate the interviewee’s involvement. In the same process as the stakeholder interviews, evaluators took notes during country interviews and then pasted notes by thematic area using a standardized template. The evaluation team also reviewed materials used to disseminate results and reports where DHS indicators were presented or the secondary analysis discussed. These documents were used to validate information from the interviews. EXTERNAL EVALUATION OF MEASURE DHS 5 C. E-Surveys of USAID Missions and Presidential Malaria Initiative (PMI) Advisors Invitations to complete an online survey were sent to 39 USAID missions and 30 Presidential Malaria Initiative (PMI) Advisors serving in countries that have either completed or have ongoing DHS surveys since 2008. The four case study countries were not included in the e-Survey. The e-Survey provided insight regarding the degree in which the DHS and MIS were meeting stakeholder needs in the remaining countries and identified major issues related to TA provision and capacity building experience and priorities. D. Review of MEASURE DHS Reports, Documentation, and Online Materials In preparation for this evaluation, ICF provided the USAID DHS Management Team with a comprehensive collection of relevant reports, documentation, and other materials in hard copy and/or CD form. The evaluation team carefully reviewed these materials and other information gathered during the team’s visit to ICF headquarters. These documents and reports verified and supplemented information gathered via stakeholder interviews and field visits. Especially useful were the trip reports prepared by ICF staff after each country visit and separate tabulations related to the intensity of TA. E. Analysis of Results The purpose of the analysis was to integrate diverse sources of information to produce a detailed and nuanced explanation of how the DHS is meeting the complex and sometimes conflicting goals of generating high quality, timely, and standardized data that engenders ownership on multiple levels and builds national capacity. The first analysis used a method of successive synthesis within, and then between stakeholder groups in order to retain their different perspectives. All interview data were stored on a single Excel workbook with separate worksheets for the five stakeholder groups and each country case study (Figure 1). Each worksheet organized data based on the Scope of Work questions. One team member took responsibility for deriving a summary of the views and experience for each stakeholder group or country, noting any diversity of views. Other team members reviewed each page and added to or queried the summarized content after reading the detailed notes. The summaries were debated during evaluation team meetings that were conducted face-to-face or virtually and a consensus was reached on the findings based on evidence from interviews. A second analysis method was used to investigate the specific question of what stakeholders valued, needed, and wanted from the DHS (Task 1A). Open coding was used to identify themes across all Scope of Work questions. The open coding was then classified according to which themes were present. The analysis methods had a number of strengths. First, summaries at the stakeholder/country group level retained the diversity of perspectives, providing insight as to how each stakeholder group would respond to recommendations. Second, the transparency of data management and decision-making across the team added to the internal and external validity of the findings. Third, the results of the e-Survey responses were available to triangulate findings from both analysis methods, minimizing the risk of over generalizing from the case studies. F. Limitations of the Methods The evaluation team necessarily focused on countries and surveys where ICF is involved. Many other countries are producing DHS-like surveys and used to receive assistance from the DHS program. These countries can be considered to have “graduated” from outside assistance, but the team was not able to make a detailed analysis of how that process was achieved and what benefits or costs were involved. EXTERNAL EVALUATION OF MEASURE DHS 6 The evaluation’s country case studies were also limited to countries in Asia and Eastern Africa. No countries in West Africa, North Africa and the Middle East, Latin America, or the Eastern Europe and Eurasia regions were visited. Data needs and survey approaches are sometimes different in these regions. For example, countries in the E&E region have focused much more on adult health and non-communicable disease than in other DHS surveys (the 2002 Uzbekistan Adult Health Examination Survey being a case in point). Innovations in survey design and field operations being undertaken in other regions could also not be assessed. For example, experience in using continuous survey approaches in Peru and Senegal could not be observed by this evaluation. The response rate for the e-Survey was 56 percent for USAID missions, yielding 22 completed questionnaires and only 20 percent for PMI Advisors, yielding only six completed questionnaires. Frequency distributions on all questions for both groups are in Appendix I (See pp. 85-96), but due to the small number of completed questionnaires, especially from the PMI group, it is not possible to draw any conclusions from a comparison of USAID and PMI responses. 5. Main Findings A. Meeting Stakeholders’ Needs Through interviews and the e-Survey, the review team encouraged stakeholders to describe what features of the DHS they valued and if DHS data met their needs. They were asked how they used the DHS results and if there were aspects of the survey content or process they wanted changed. This section is divided into three sub-sections. The first sub-section describes what value each stakeholder group sees in the DHS and the extent to which the DHS meets their needs, including those that some informants believed were not being addressed adequately. In the second sub-section, we summarize views on the strengths and weaknesses of several stand-alone surveys that were implemented during the MEASURE III funding round: AIS, Malaria Indicators Survey, and SPA. The third sub-section addresses higher order needs and aspirations for the DHS. We analyze what stakeholders see as the appropriate goal and objectives of the DHS program and consider whether these higher order needs are manageable in its current configuration. Stakeholder needs and expectations specifically related to TA and capacity building are addressed in following sections. EXTERNAL EVALUATION OF MEASURE DHS 7 Needs that are met Approach taken to meet need Costs or shortcomings of the approach Benefits of the approach High quality population based data Close adherence to a sequence of best practices in cross￾sectional population survey methods Requires a high level of technical expertise Many of the quality assurance measures are expensive The DHS results are very highly regarded Internationally comparable indicators for monitoring US and global programs Use of a core questionnaire to produce standard indicators, engagement with expert working groups Present approach depends on professional goodwill and does not have a recognised governance process to define indicators and technical issues, raising the risk that decisions can appear arbitrary. The DHS program is able to respond to emerging priorities such as malaria eradication and reduction of maternal mortality Data availability The DHS program has a web-based compiler to make it easy to produce indicators over time and between countries. Recoded data sets are available in a number of formats Population and health surveys done outside of the DHS program are not easily incorporated into STATcomplier or made into standard recode data sets The information is available to the international global health community leading to increased transparency and a resource to generate new knowledge Host-country needs for population and health data Extensive consultations and use of stakeholder workshops, Steering and Technical Committees Incorporating host country data needs increases pressure for expanding the length and sample size of the survey which leads to greater operating costs and logistical demands Great satisfaction with the data produced EXTERNAL EVALUATION OF MEASURE DHS 8 Stakeholders Needs Met by the DHS Global Stakeholders Stakeholders responsible for global programs such as senior officers at USAID and the Global Health Initiative and multilateral organizations such as UNICEF, the World Bank, and the United Nations Population Division expressed considerable satisfaction with the DHS for providing high quality national level information. For them, the strength of the DHS is in the use of consistent methodologies that produce comparable and internationally available data. They frequently used the term “gold standard in health and demographic data” to describe the DHS and some informants reflected that the program was one of USAID’s greatest initiatives. DHS results are often consulted in preparing final estimates and projections of global population and health conditions. The number of research reports using DHS data at meetings of the IUSSP, PAA, the Global Health Council, and other professional meetings has been impressive. The availability of standard recode datasets in several formats has greatly increased the utilization of DHS materials around the world. This stakeholder group particularly valued the StatCompiler data retrieval system because they need to report on cross-sectional indicators from multiple countries. Global stakeholders involved health intervention programming also spoke highly of the DHS and valued its quality, comparability, and access. They or their offices often used standard recode data sets and asked for special tabulations. In addition, these informants had worked with ICF staff and praised their technical skills and willingness to incorporate some of their specific data requests; however, this group also had several specific data requirements that were not being met. These needs were often related to program-specific indicators, the need to disaggregate information to sub-national levels, or the need to provide more timely results. The stakeholders involved in health programming were also interested in methodology. They looked to the DHS for guidance on advancements in measuring health behaviors using new study designs such as continuous or panel surveys or the use of new technology. Some informants who raised these expectations said they were disappointed that MEASURE DHS was not more aggressively involved in promoting such innovations; however, they did note that MEASURE DHS has been taking steps to better document DHS survey methodology through the release of the DHS Toolkit. Country Level Stakeholders In-country stakeholders praised the DHS for the same reasons as the global stakeholders. They told the evaluation team that DHS data for their country was the most authoritative data available due to the involvement of international experts, standardized indicators, and comparability with other countries. Country stakeholders also appreciated that the DHS was owned by the government and designed to meet its needs. The DHS project has institutionalized systematic steps to ensure that a wide range of relevant host country stakeholders are routinely involved in the identification of data needs and questionnaire design. This approach has resulted in widespread support of the surveys, which the team found through the case study countries and e-Survey. While the DHS core questionnaire and specialized survey modules are initially designed for consistent cross￾national specifications, ICF encourages in-country stakeholders to submit additional requests for information. The formal mechanisms for engaging stakeholders is the high level Steering Committee which is usually chaired by a senior person from the Ministry of Health and a Technical Working Group that addresses the practical issues of survey design, content, implementation, and dissemination. The Technical Working Group consists of representatives from donors, local implementing agencies, line departments within the Ministry of EXTERNAL EVALUATION OF MEASURE DHS 9 Health, and where relevant, researchers and local and international NGOs. Both committees usually meet regularly and ICF technical advisors attend when they are in country. When the survey is being designed, the ICF country manager will usually meet individually with key stakeholders to discuss their needs and then hold a large workshop. Although every suggestion is treated with respect, ICF discourages the adoption of unworkable ideas. Behind the scenes, there are disagreements between different government agencies that on occasion, can only be resolved by a Minister or in one case, the President. Although it has not always been possible to please everyone, the DHS instruments that emerged from preparatory discussions with partnering countries reflect a broad consensus on what information is needed. Even informants who told the evaluation team they wanted different questions in the DHS were satisfied with the decision-making process. The e-Survey findings confirm the overall support of ICF’s inclusive approach to managing country needs. USAID missions report that the DHS has been very responsive (52.4 percent) or responsive within technical or budgetary constraints (42.9 percent) to the information needs of stakeholders. The major stakeholders who were heavily involved in developing the recent DHS were typically Ministries of Health, Ministries of Planning and Finance (where LIAs for the DHS are often located), and UN agencies such as UNFPA, UNICEF, and WHO. Local NGOs were usually not identified as major participants in specifying the content and operational plans for the DHS. USAID missions typically use DHS information for the monitoring and evaluation of demographic impacts and program outcomes, program and project formulation, policy dialogues, and internal reporting requirements. Findings show that 91 percent of USAID missions’ use the final country report and 76 percent refer to summary reports while 43 percent employ survey data files and 33 percent consult the DHS StatCompiler. DHS data that are considered essential by missions tend to vary by region with HIV/AIDS and malaria more commonly cited in sub-Saharan Africa. In the four countries visited by the evaluation team, the data were widely used. National development plans in all four countries use DHS-derived indicators. In-country stakeholders mentioned that when there were alternative estimates for the same indicators, DHS measures were preferred and selected. In addition to reporting national indicators, Ministries of Health built DHS indicators into their reporting and planning processes at sub-national levels. Governments, donors, NGOs, and research institutions in all four countries have also taken initiatives to produce publications highlighting summary findings and policy implications, with or without the financial or technical support of USAID and MEASURE DHS. EXTERNAL EVALUATION OF MEASURE DHS 10 Unmet Data Needs Identified by Stakeholders Needs that are not met or could be met better Current approaches Approaches that could be taken to meet need Costs entailed in adopting the approach Benefits of the approach Greater country leadership in all phases of the survey continuum Involvement of Steering and Technical committees Use of resident advisors, national and regional consultants Formal processes for discussing and monitoring TA input Greater use of tools that enable local implementing agencies to undertake more data processing on their own Development of a dissemination and further analysis plan at the beginning Funds may not be available to meet identified needs Delays may be experienced during fund raising, survey implementation or report writing Greater planning will result in more targeted assistance and a common purpose for capacity building More capacity to use data for planning, managing and monitoring health programs and secondary analysis Currently efforts are ad hoc Some of the most commonly used are workshops for data usage and the DHS Curriculum The DHS curriculum model appears to be more effective because it builds up the strength of institutions. Local institutions are better able to meet the demand for ad hoc requests for data training MEASURE DHS not currently funded to have long-term relations with national institutions. This model would require some continual engagement by the US contractors with local institutions. Capacity building efforts will be more effective. More local capacity to manage data processing phases Current efforts are very dependent on TA While some external TA may always be desirable, more user friendly tools would assist LIA to do more on the DHS but to also use the tools for other surveys Development of tools is very technologically challenging and time consuming. Not clear if this is desired by LIA. The demand should be tested before embarking on this track. Reduction in the cost of external TA. More policy relevant outputs Ad hoc support for ICF involvement from USAID Missions Using the design phase as a time to develop outputs. On-going support for preparing organizations to be able to use the data as soon as it is available. Not a clear funding stream for the production of local materials, but this is unlikely to be expensive. Greater continuity between data production and use. EXTERNAL EVALUATION OF MEASURE DHS 11 Country level informants also identified unmet needs from the DHS, other than those related to TA and capacity building that will be discussed later. The unmet needs relate to the frequency and timing of surveys and the results and greater use of data for policy and planning. Dissatisfaction with the frequency of surveys and the timing of the release of results were two concerns that came up at the national level and between U.S. and global stakeholders, less commonly. In the e-Survey of USAID missions, “excessive” delays between the completion of fieldwork and the release of final reports was noted by five out of 22 Missions surveyed. We heard similar complaints in case study countries that results were not available sooner after fieldwork. ICF aims to produce a final report within 12 months of completing fieldwork but slippage can occur for a number of reasons. Preliminary results of key national indicators are presented earlier but some stakeholders have particular needs that require additional results, which they report ICF is reluctant to provide due to the sequence in which they do tabulations. Fieldwork can be delayed for a number of reasons including lack of funding, equipment that failed to arrive, or conflicting demands on the local implementing agency (LIA). Recent surveys in each of the case study countries commenced later than originally planned. For these unavoidable reasons and the regular cycle of surveys, government and donor planning cycles did not always correspond well with the arrival of DHS information. In addition to national data requirements, at least one person in every case country explained to the evaluation team that their next survey should be done at the last possible moment to have current Millennium Development Goal indicators in September 2015. In Cambodia, the DHS data were thought to be so essential for health planning that the production of a five￾year plan was delayed until indicators were available. The entire health budget had to be tried as an extra budget item for that year while the planning documents used for national budget allocations were in limbo. Similarly in Malawi, information was required from the 2010 survey to plan a new health sector strategy but the final DHS report came out too late; instead, reliance was placed on other information including older DHS findings, facility-based evidence from the Ministry’s HMIS, and results from UNICEF’s Multiple Indicator Cluster Survey (MICS). Policy and program relevant dissemination was another commonly mentioned unmet need at the country level. Final reports were seen as important, but of limited use because they were too complicated and sometimes lacked national contextual information. Felt needs for relevant output ranged from dissemination at the sub-national level (the administrative locus for many decentralized health systems) to special factsheets, handbooks, and policy briefs and the training journalists. Most informants added that these products had to be produced in the local language. In general, the view was that local experts could produce many dissemination materials, but financial and technical support through MEASURE DHS was welcomed. Closely related to the need for policy relevant information was the desire for the DHS to provide district or other sub-national information. With decentralized health services being a feature of so many countries’ health planning and service delivery systems, data is relevant only if it can describe what is happening at the district level where decisions are made. This need, which is being met by the MEASURE DHS program in some countries with funding from other donors, is driving the larger sample sizes in this funding period. For example, the sample size for Indonesia’s last DHS in 2007 was increased to 32,895 to allow for the estimation of results for all provinces in the country. Findings Specific to the Non-Standard MEASURE DHS Surveys MEASURE III DHS has developed or refined stand-alone surveys to meet specific stakeholder needs. The degree to which these have been employed and their success in meeting the demands for information vary. EXTERNAL EVALUATION OF MEASURE DHS 12 SPA The facility-based Service Provision Assessment Survey (SPA) has been implemented 15 times in 11 countries since 1997. In Phase III, it was conducted in Kenya and Namibia and is in planning stages in Malawi, Senegal, Nepal, and Tanzania. ICF worked with the WHO to harmonize facility-based tools for monitoring health system strengthening. Revised instruments based on work by ICF, independent consultants, and a pretest conducted in Kenya was submitted to USAID in late 2011. The SPA is a complex instrument that consists of an inventory of infrastructure and supplies, observation of clinical processes, exit interviews with patients, and health worker interviews. The revision halved the number of core inventory questions. These changes were in response to concerns from stakeholders in Washington D.C., Bangladesh, and Tanzania: the SPSs were too difficult to implement and analyze, resulting in underutilization of results. There is little doubt that there is a demand for information on facility-readiness despite evidence to suggest that SPA results have been underutilized in some countries. The streamlined instrument may result in an increased use of SPA data. It will be tested in Malawi in 2012. AIS The AIDS Indicator Survey (AIS) was designed as a stand-alone instrument to provide indicators for monitoring the President’s Emergency Plan For AIDS Relief (PEPFAR) and UN General Assembly Special Session on HIV/AIDS (UNGASS). In MEASURE III DHS six AISs have been completed or are ongoing, including a combined AIS and MIS survey in Tanzania. One reason for the small number of AISs is that HIV/AIDS is viewed as a key reproductive health issue that should be addressed in the DHS standard (core) questionnaire. Questions related to knowledge, attitudes, and risk factors for HIV have been integrated into most DHS surveys. HIV seroprevalence estimation using dried blood spots are also being incorporated into the standard questionnaire at the request of some governments although ICF discourages this practice for countries with a HIV prevalence of less than two per cent. In many cultures, a general household survey may not be the most effective vehicle to collect information on premarital sex and concurrent sexual partners. Informants from the two Asian case study countries (Bangladesh and Cambodia) expressed these concerns. In Cambodia, the CDHS probably underestimated the degree of premarital sex occurring among young women. In Bangladesh, a small study with highly trained interviewers uncovered a much higher rate of out-of-marriage sexual behavior than was estimated from a conventional household survey. The Tanzania AIDS Commission and the UNAIDS M&E advisor in Cambodia both spoke highly of the information from the AIS. In Tanzania, AIS results are used extensively for program planning and evaluation; although, some informants wanted the information collected more regularly. They are also using AIS results to identify high HIV transmission areas in which to allocate prevention and treatment services. The HIV seroprevalence estimates from the Cambodia 2005 DHS created some disputes with various stakeholders about whether prevalence estimates based from sentinel antenatal clinics was “better” than population-based estimates. MIS The Malaria Indicator Survey (MIS) provides key indicators for the PMI. ICF participates in a multiagency group, which advises on malaria indicators. Twelve stand-alone MIS surveys have been implemented during MEASURE III. The survey measures malaria prevalence and the effectiveness of interventions such as malaria drug therapy, domiciliary insecticide spraying, and the use of insecticide-treated bed nets. The evaluation team heard a great deal of praise for ICF’s work in malaria testing, especially with respect to using EXTERNAL EVALUATION OF MEASURE DHS 13 rapid diagnostic testing. ICF has provided TA in malaria testing for surveys organized by other funders such as UNICEF’s Multi-Indicator Cluster Surveys. The international malaria community is said to be “in a hurry” to collect information every two years using the MIS. Its major strength is that it standardizes questionnaire design, data collection, and analysis procedures so that results are consistent with methods employed in other countries. According to an informant in Tanzania, the DHS brand also adds to the broad acceptance and credibility of survey findings. Dissatisfaction with the survey reflected the challenges of conducting surveys during the wet season, the highest period of malaria transmission. Out-of-season data collection is more likely to happen if the MIS is merged with a much larger DHS. Stakeholder Expectations: The Big Picture In addition to detailed comments regarding the present delivery of DHS, global and in-country stakeholders expressed their views on what the DHS should be. These aspirations reflected both technical and strategic issues. There were three main themes: concern about the surveys being too long, the view that USAID and other global and in-country stakeholders were relying too much on surveys at the expense of other forms of health data, and the need for greater in-country leadership in all stages of the survey process. Too Long and Complex Global stakeholders from USAID, other parts of the U.S. government, and external organizations expressed concern about the size of the questionnaires. An average DHS core questionnaire now contains around 600 questions, down from over 800 questions at the end of the last funding phase. With many countries electing to add DHS modules that integrate HIV/AIDS or malaria indicators or collect country-specific information, the average number of questions in this phase has climbed up again. During the 1980s, 100 percent of all respondent interviews were completed in 45 minutes or less. At the present time, only 30 percent of all interviews are completed within 45 minutes. The time interviewers spend conducting household and women’s and men’s surveys and biomarkers are not readily available, but informants from ICF and in-country estimated that with the household and women’s and men’s surveys and biomarker collection, interviewers could be with a family for up to four hours. In addition, with the growing demand for increasingly disaggregated data, the average sample size for the household surveys has risen from an average of 8,119 during the 1980s to 18,714 for countries with four or more DHS surveys. This growing complexity poses challenges in effectively managing survey field operations and ensuring the timely release of high data quality. The length of the survey is generally seen as a consequence of the success of the DHS. Stakeholders who expressed concern about the size were so supportive of DHS content that almost no one made suggestions of what could be taken out. Very few informants mentioned design solutions, although a few suggested only asking some questions of a sub-sample of selected households. In the four case study countries and many trip reports from other countries that were reviewed as part of this evaluation, it was clear that ICF actively advised keeping the survey to a reasonable length but the process of stakeholder engagement and multiple funders prevented the adoption of this general advice. Over-Reliance on Surveys A number of informants were concerned with what they perceived as an over-reliance on population survey data within countries and as part of USAID’s overall funding strategy. One or two stakeholders in each case study country expressed this view, typically representatives of donor organizations, including the USAID missions who took a whole-of-sector approach. EXTERNAL EVALUATION OF MEASURE DHS 14 Five out of the 21 stakeholders interviewed from USAID (outside of the DHS management team) and other U.S. government agencies also expressed concerns that a re-balancing of data investment was required. The rationale included the rising costs of the DHS survey, but was primarily based on the perspective that health program monitoring needed other instruments such as HMIS, health surveillance, and small surveys targeting special populations or service catchment areas. Malaria, tuberculosis, and health system strengthening were particular areas where process and utilization data are not easily monitored through national population surveys. The contractor is also very aware of the limitations of population surveys. ICF’s review of procedures for the testing of malaria presented in the 2011 Award Report concluded that well-functioning health information systems would provide more meaningful information on the prevalence and incidence of malaria parasites than a survey. Greater National Involvement and Ownership Many stakeholders said they wanted to see greater involvement of national governments and institutions as well as local experts in the DHS. Some respondents, especially those in country, wanted greater host country involvement in all aspects of the survey process from design to analysis and report writing. They told the evaluation team that development assistance would eventually end and they needed to be able to conduct health surveys on their own. For example, in the e-Survey of USAID missions and PMI representatives, 10 out of 16 people who gave open-ended responses to a question about what they wanted in technical advice for the next survey indicated they preferred a more significant role played by national institutions. In most, but not all, cases this included a reduced role for the U.S. contractor, but in some cases it involved greater efforts by the contractor to build capacity. Other stakeholders, particularly those with regional or global perspectives, argued that the objective of the DHS should enable countries to “graduate” from the DHS program and be able to conduct household surveys themselves. Changing the methods of TA and placing greater emphasis on capacity building activities were suggested solutions. B. Effectiveness of Technical Assistance Models of TA Definition Views or Evaluator Comments (if new) Costs/benefits Full 10-12 visits from ICF Calverton-based staff Credible, knowledgeable. TA is oriented towards timely delivery of quality data, and also walking local partners increasingly through stages of the survey implementation as key partners. TA should lessen over time. USAID/ICF should recognize in￾country capacity in areas where appropriate. Some countries want more TA, or more time spent in-country for capacity-building aspects of the work. Limited Countries with significant capacity receive less ICF visits Visits specific to sampling, data processing, and report writing were requested across both “limited” and Full-package TA environments. The TA specific to data processing needs investment and greater consideration in future DHS designs and strategies. EXTERNAL EVALUATION OF MEASURE DHS 15 This section describes the TA tasks currently provided for a DHS, presents the evaluation team’s findings on the effectiveness and efficiency of the TA provided, and reviews the advantages and disadvantages of current TA models. Technical assistance models and tasks The typical DHS has a very large TA component. ICF estimates that their “full” package of assistance equates to more than 1.5 person years of effort from the first discussions about survey design to national dissemination. The limited model has approximately 40 percent fewer person-days. ICF estimates that out of 38 surveys started or completed in the MEASURE III DHS as of 2011, only seven used a limited model. The limited model is employed when local implementing agencies are very proficient. In two countries, the DHS was merged with the UNICEF MICS and the majority of TA was being provided through other means. Key informants from all stakeholder groups identified factors which increased the need for TA over time; these include new modules and technologies, greater logistic and data quality challenges with larger sample sizes, and high turnover of experienced staff in local implementing agencies. The actual level of TA delivered is determined during the first design visit to the country by the regional coordinator or other senior staff member; this visit is funded by USAID/Washington. Field support funds, usually from the USAID mission, pay for the remainder of TA costs. Details about the level of TA required, including country visits, are decided in discussions with the USAID mission and in theory follow an assessment of national capacity influenced by ICF’s previous experience. The evaluation team was told that there were no detailed discussions in the case study countries about the level of TA required when the most recent surveys were planned. In trip reports, the only reference to TA needs at the planning phase related to sampling and biomarkers. According to ICF, TA is frequently adjusted upwards during survey implementation in response to unforeseen needs. Strengths and weaknesses of the TA model Many stakeholders praised ICF’s technical advisors, telling the evaluation team that these individuals were friendly, expert, hard-working, and willing to share knowledge and skills. The ICF staff and consultants interviewed were proud of their work with national counterparts. For example, respondents in Tanzania praised the TA, claiming that it targeted areas of weakness and contributed to building capacity. Tanzania was viewed by ICF staff as a success story, with the National Bureau of Statistics gaining considerable skills in survey administration. The near constant schedule of DHS surveys in the country may have been a factor in ensuring continuity of relationships between national and ICF staff. The role of ICF in facilitating survey design and content was highly regarded. Assistance with sampling is also appreciated in all settings because this is a specialized skill and plays a vital role in data quality. Even where capacity is high, an ICF sampling expert reviews the sample design before data collection proceeds. Technical assistance for data processing was viewed more critically in-country. Data processing encompasses many tasks from creating the data entry and field-testing tables to producing tabulations for the final report. Technical assistance in data processing include training LIA staff on how to do routine tasks like data keypunching and CSPro programming for data entry, consistency checking, and other diagnostics of error patterns. After working with ICF, most LIAs develop some capacity for employing standard data entry and basic editing checks in CSPro. However, many LIAs in our four case study counties expressed a keen interest in becoming more adept in the use of CSPro. ICF pointed out that because some aspects of programming were so specific to the DHS surveys, it was improbable that officers at a LIA could become proficient after only one survey. They estimate that it would take two years to train someone with a background in programming and statistics to proficient in all data processing tasks. EXTERNAL EVALUATION OF MEASURE DHS 16 Few LIAs have the capacity to produce tables or complex calculations required for weighting and creating certain indicators such as maternal and child mortality. Several contacts told the team that when they enter other surveys in CSPro, “they export the data to SPSS as soon as they can” and do the analysis from that program. ICF staff or consultants must play a major role in the tabulation process in most countries. The recent move by ICF to produce standardized programs in SPSS and STATA for generating tables and indicators could reduce the need for TA. For example, the private research company contracted to conduct the survey in Bangladesh is able to do many data processing tasks, but needs hands-on assistance in advanced programming necessary for tabulations. A local professor of statistics was hired as a consultant, but both parties reported that they did not have the opportunity to do the programming for themselves with the support of external TA. In Cambodia, the National Institute of Statistics was very pleased with their achievements in conducting thorough secondary edits, but they recognized they did not have the capacity to do the programming for consistency checks and tabulations using CDHS data. They wanted their programmers to be trained so they could work more closely with external advisors. Several stakeholders in Malawi told the evaluation team they were frustrated and embarrassed that many of the tabulations for their final report were done in the United States. Scientific report writing is considered a weakness in most countries and not only in those that are non￾English speaking. ICF’s current model for report writing is a two-week in-country report-writing workshop facilitated by the country manager for the survey. These workshops bring people together from many national institutions and NGOs, increasing the number of people who are trained to understand and interpret DHS indicators and tables. This model is generally considered to be more cost-effective than the former approach of bringing small numbers of nationals to ICF headquarters for 4-6 weeks to review pre-prepared drafts. During these workshops, national level stakeholders are given opportunities to help interpret data and contextualize it per services provided in that country. The workshops also increased the number of people who understand how to interpret indicators and assist their agencies in applying the results to health programs. Present and future technical assistance models Technical assistance for the DHS is primarily provided as short-term visits and some remote work by full time staff from ICF. Other models have the potential to reduce TA costs without sacrificing quality and contribute to broader capacity building goals. Any change of the current TA model should include documentation of the amount and type of TA provided by survey and country and for the full MEASURE DHS program. South-to-South TA A TA model, which ICF currently uses, creates a pool of experienced people based around the world who are employed by ICF as consultants to perform short term work in-country and sometimes in the U.S. Many of these individuals are DHS “alumni.” They had worked, and sometimes continue to work, for LIAs. They typically gained their skills as participants in the survey process and took up opportunities for more intensive training offered through the DHS program and applied their skills to other surveys conducted by their agency. This pool of workers enables ICF to support increasing numbers of surveys. Since they are based in the regions where DHSs are often conducted, professional fees and travel costs are lower and the consultants frequently know the local languages. For example, in the Bangladesh DHS, a Bengali speaker living in Nepal was hired to assist in fieldworker training. In-country case study stakeholders spoke highly of this form of south-to-south TA. Such consultants bring the necessary skills and may have time to give more on-the-job training. At present, these consultants are principally used for fieldwork training, supervision, and data EXTERNAL EVALUATION OF MEASURE DHS 17 Significant expansion of south-to-south TA will require greater investments in identifying and training people with the necessary technical skills, including spending time at ICF headquarters and accompanying more experienced ICF staff on country visits. These regional experts would also require institutional affiliations that enable them to take up short-term consultancies. At present, the nurturing of these professionals is done partly as an investment by ICF and partly by taking advantage of USAID/Washington funded regional training programs. Future MEASURE DHS contracts could stimulate the provision of south-to-south TA by requiring U.S. contractors to have regional partners responsible for providing local experts for this purpose. The savings in TA costs from the U.S. could make this a sound investment with no loss of survey quality or independence. National Residential Advisors and Consultants In many countries, including Cambodia and Malawi, ICF employed a full time local or regional consultant to be based in the national statistical office. A NSO may lack someone who can be assigned as a full time survey coordinator so the hiring of a consultant assists with the management of the survey. This person is typically a local researcher with good ties to the NSO or MOH and experienced with previous DHSs. This person is usually involved in training and fieldwork supervision or quality control and rarely has data processing skills, but does monitor progress. Resident advisors are used in addition to a full TA package rather than as a replacement for external TA; however, ICF reports that a resident advisor may reduce the need for visits or shorten the length of TA visits, thereby offering some savings. Local researchers such as statisticians, demographers and public health researchers are frequently employed to work on some aspects of the DHS. In some cases, these people provide an important service for the LIA or the Ministry of Health. They can provide timely advice and serve as an intermediary between different national bodies or between the government and U.S. contractor. However, the evaluation team also heard cases of local experts being unsure of their role and not given much opportunity to apply or increase their technical skills. In contrast, where local NGOs or research firms were contracted to undertake dissemination work, the feedback was usually very positive. These different experiences suggest there are lessons that can be learned on how to get the greatest benefit from local consultants. For dissemination activities, local consultants have an obvious advantage in that they know the language and understand the context of national health programs. If not already known by ICF, the added benefit that local technical consultants bring may not be recognized and are therefore not utilized. C. Task 2: Effectiveness of Capacity Building Activities Based on stakeholder feedback, the Request for Proposals for MEASURE Phase III required a greater emphasis on capacity building than in MEASURE Phase II. Capacity building is integral to all aspects of the survey process. The contractors were expected to incorporate individual and institutional capacity building goals in all stages including planning for population and health data needs, elements of data collection and processing, targeted data dissemination to maximize use, and increasing further data analysis. Many stakeholders in USAID missions, U.S. government agencies, and international organizations said that they expected to see greater progress in countries being able to do the surveys themselves without the present levels of assistance from ICF in current rounds of DHS. ICF informants generally supported the goal of capacity building with one or two exceptions, but noted that capacity building is a long-term process that may actually require higher levels of TA in the short run to enable countries to develop the capacity to take on more responsibility for future surveys. In-country stakeholders including USAID missions and multilateral donors, local implementing agencies, Ministries of Health, researchers, and NGOs all prioritized capacity building. Ninety percent of e-Survey respondents noted that ICF has built capacity of local implementing partners and individuals to plan and EXTERNAL EVALUATION OF MEASURE DHS 18 conduct surveys. A significant number (61.9 percent) also acknowledged ICF’s work in “increasing skills among multiple audiences to interpret and use DHS results for evidence-based planning and monitoring.” In the four case study countries, almost every stakeholder was able to indicate areas in which local capacity needed to be improved and expressed a desire for the next DHS to be conducted differently to enable a greater hands-on role for national institutions. Instead of seeing capacity building as a trade-off with quality, they saw local production and analysis as a primary indicator of quality. ICF produced a Capacity Building Strategy for the Year II Award report. The objectives of capacity building are: • Improving the skills and self-sufficiency of implementing agencies and individual staff members to plan and conduct surveys and collect, process, and disseminate data • Increasing skills among multiple audiences to interpret and use DHS results for evidence-based planning and monitoring • Enhancing skills among researchers and academics to analyze DHS datasets The strategy also states that building this capacity is not feasible in most countries because of funding constraints. In MEASURE III DHS, ICF has undertaken an impressive range of capacity building activities (see above Table) that can be viewed as capacity building to address supply and demand of the data circle. This evaluation describes and reviews each type of activity and comments on the evidence (or lack thereof) of the effectiveness of these activities. Based on our stakeholder interviews in case study countries and e-Survey results, these capacity building activities are highly valued and there is a large unmet demand for more of these types of activities. None of these activities were regarded as poor or less effective vehicles of capacity building. There are many examples of growth in national survey capacity. Informants point to many countries that “graduated” from full TA to managing their own population and health survey programs or forming collaborative relationships for TA on elected areas. Egypt, Turkey, Honduras, Colombia, Peru, the Philippines, Thailand, and Indonesia are among the countries mentioned. On the ground, country informants were also convinced that capacity was being developed. All of the 22 USAID missions and six PMI officers responding to the e-Survey said that they had seen some growth in national capacity, mostly in the ability to conduct population-based surveys. Unfortunately, there has been no systematic effort to collect information about capacity growth although some outputs have been tracked. Part of this knowledge gap about the DHS is related to funding and planning of DHSs. Most funding for capacity building that falls outside of OJT is mission-based and not centrally based. Because of this and other considerations, the program does not routinely report as a whole on the “status” of countries along a continuum of capacity areas or strengths. However, a more powerful explanation for the lack of documentation is that there is no consensus on the goal of capacity building. MEASURE III DHS suggests that countries should be able to conduct their own surveys, but this goal is undermined by the development of new modules and technology. At the country level, work plans include capacity building activities but these are not based on a formal assessment of capacity and do not incorporate measurable targets or outcomes. There is tension regarding how much of the current MEASURE DHS can realistically be expected to be involved in capacity building. Current capacity building activities EXTERNAL EVALUATION OF MEASURE DHS 19 As documented in the Capacity Building Strategy, ICF has proposed and undertaken a range of different capacity building activities during MEASURE III DHS. They can be viewed as building capacity for implementing high quality surveys or for improving data utilization and analysis. Each type of activity is reviewed here along with an assessment of the evidence pertaining to the effectiveness of these activities. Implementing High Quality Surveys On-the-job training Persons who are provided on-the-job training are all assigned to work on the survey, including implementing agency personnel, supervisors, editors, survey managers, report writing staff, etc. ICF personnel work side by side on an ongoing basis to ensure that the job they have been assigned to do in the DHS is done to the highest standards. ICF finds that on-the-job training requiring hands-on learning experiences is a more effective approach for creating learning that can be applied to future similar surveys than formal classroom training. On-the-job training is not sufficient for increasing institutional capacity unless it includes structured training fitted into the implementation schedule. In Cambodia a hard-working two-person data processing team from ICF was able to free-up time during the last week of their second data processing visit for more structured training of the national statistical office. This opportunity helped staff to gain a much greater understanding of how the process sequence worked. However, it occurred too late to be of practical use for the current DHS and there is little scope for them to apply what they learned until the next DHS is scheduled in five years. Scheduled formal classroom training for operational skills necessary for the implementation of the survey – interviews, field editing and quality checking, lab testing and recording, data entry and managing the resolution of errors, is part of the routine implementation of population-based surveys. Because people who do the work must be taught in an organized fashion how to do tasks correctly and because those skills are immediately applied, the capacity gains are tremendous. The formal training also provides a way for individuals who performed especially well in a previous study to gain work and greater expertise by being involved in training others for the next survey. The evaluation team heard many examples of individuals who gained experience this way. Most informants said that LIA’s had developed the capacity to do these key aspects of population-based surveys. Mentoring for advanced skills is another effective capacity building technique. Individuals with aptitude and drive, and perhaps advanced education can gain more technical skills in data processing through ongoing mentoring by distance along with on-the-job training during the survey process. It is possible for in-country national staff to prepare the first draft of programs or a sampling design and send it to ICF headquarters to be reviewed. This increases the responsibility of national staff while still maintaining the benefits of the contractor’s expertise. This method is usually done either at the insistence of the national statistical office or, more commonly, when there is already a relationship between the LIA and ICF staff formed during previous surveys. Sometimes these individuals have an opportunity to spend time at ICF headquarters to work on a previous survey or attend formal advanced training. Several people mentioned a programmer from Turkey who came to ICF headquarters to work on another survey in exchange for advice on a Turkish health survey that was done outside the MEASURE DHS program. The use of local consultants as fulltime advisors is a form of capacity building. These highly motivated and skilled individuals advance through their job responsibilities and close relationship with ICF. One grateful senior MOH officer in Cambodia named the local resident advisor as the best example of capacity building from the DHS. She said ‘we have succeeded to train one Bunsoth [the local consultant], now we need five more Bunsoths.’ EXTERNAL EVALUATION OF MEASURE DHS 20 Formal classroom training for more advanced skills is highly contingent on funding availability. Regional training can be supported through USAID Washington, but in-country training is paid through an extra budget line in field support. To be useful for the current DHS, the training needs to be planned and delivered prior to the start of the survey, understanding that flexibility is required during the survey process to ensure that individuals who will be doing to work can participate and should “new” funds become available trainings may be planned mid-stream. Bringing nationals to ICF headquarters for intensive work experience, particularly data processing, also needs to be funded from field support. Selecting the right people to attend formal training is a perennial problem. To be effective, participants need a strong foundation of knowledge and to be in a position to apply what they learn. Without a competitive selection processes, there cannot be any guarantees that the right people will attend. The compilation of the DHS Toolkit was an award criterion for this funding period. The Toolkit is designed to help organizations conduct population-based surveys on demographic and health topics. While many of these Toolkit documents have been available for some time on the DHS website and/or in printed form, the Toolkit is the first time that all these materials will be available in one accessible place. The goal is to provide a “step-by-step” package of materials from the initial survey design to dissemination of results. There is good synergy with the Tool Kit and the DHS Curriculum. These resources have been used to teach selected African researchers and is already being used in universities and training programs. Short-term fellowships are offered to selected young researchers through USAID Washington funds. Training takes place at ICF headquarters or another country for fellows from the surrounding region. These fellowships usually have a specific objective of completing a research paper. They have proved to be time￾intensive for fellows and staff who are required for one-on-one mentoring. It can be a challenge to offer formal training suitable for a widely diverse range of skills. Other outputs such as being able to deliver the DHS Curriculum may be more effective. There is also increasing interest in making fellowships more competitive to ensure that participants come from research environments conducive to applying their skills and rewarding their achievements. ICF proposed a redesign of the fellows program in Phase III to focus on in-country junior faculty. Through the redesign, ICF Macro would establish institutional capacity among universities, starting in Africa, to conduct international-level health or demographic analysis using DHS data. Kenya was proposed as the site for the first test of this approach. Eight to ten Fellows will be selected. The Fellows will prepare research papers with ongoing feedback and support from ICF Macro specialists. Improving Data Utilization and Analysis Data utilization training is becoming an expected part of the DHS process, ideally occurring immediately after the national dissemination workshop when interest in results is high. Often these workshops have a train-the￾trainers element because participants are responsible for disseminating results outside of the capital. The DHS Curriculum includes instruction on all stages of survey design and implementation, but its focus is on the interpretation of indicators and their use in policy-making. The Curriculum is African-specific, but could easily be adapted for other regions. In this funding period, ICF has conducted six core curriculum trainings, two entirely delivered by local researchers. ICF sent follow-up evaluations to 49 participants in the core-funded trainings in Kenya, Tanzania, and Uganda. Among the 26 participants who opened the e-mail, 16 (33% of entire sample) said that they had used the curriculum five or more times since the training. ICF plans to do another survey of participants of subsequent trainings in a few months. EXTERNAL EVALUATION OF MEASURE DHS 21 Other training is happening outside the formal DHS mechanism. For example, in Cambodia UNAIDS held a two-week data analysis workshop with an external consultant. Participants came from the government’s HIV/AIDS line agency and reproductive health NGOs. Special purpose publications, such as policy briefs and fact books based on DHS data are increasingly being produced in country rather than in Calverton, Maryland. ICF typically becomes involved when the mission allocates funds for these activities. International and national NGOs and UN agencies often take data from the final report or their own secondary analysis to inform audiences about health issues or advocate for policy change. Global access to the data facilitates this use, but misinterpretation of indicators is possible if there is not a pool of sophisticated data analysts familiar with the DHS. Well-targeted data usage workshops can minimize the danger of inappropriate use of data. Report writing workshops. Report writing is seen as an area of weakness in almost all countries. In the past, ICF brought individuals from ministries of health and LIA’s to headquarters for 4-6 weeks. They reviewed report drafts, prepared or updated country-specific background material and had input into other survey outputs such as key findings. The current model of a two week writing workshop in-country is preferred. More participants gain in-depth instruction on how to read and interpret DHS tables and, with the support of facilitators, write or review first drafts of the report. The reports are finalized, subject to national government approval, at ICF headquarters. Most stakeholders reported that this was a better process. It increased the number of DHS ‘champions’ and results in reports which were more relevant for the country. However, several of the reports produced in workshops still needed extensive revision and editing by the ICF team. Journalist training. Additional funding from the Mission in Bangladesh enabled separate training for journalists to use DHS data in health reporting. Follow-up has shown that there has been some increase in the amount of health reporting following the training. Additional work with journalists is funded and underway in Bangladesh. In order to make these efforts more sustainable a handbook on health journalism is being prepared for journalist and an online course for journalism students. These initiatives in Bangladesh and Africa involve national, regional and international health media experts. Secondary data analysis training. Three workshops were completed in Indonesia, a workshop in Albania and others in Uganda and Cambodia. These are open to participants from selected national institutions and NGOs and frequently include instruction of statistical software packages for data analysis. To be most effective, these secondary analysis training sessions need to be properly planned, budgeted, and technically backstopped. Without clear expectations from their institutions, participants may not apply the knowledge they have gained. Making publications and data sets fully available through the website, including the flexible STATcompiler, is a widely valued capacity building service. So far the greatest demands for these services have been from donors and foreign universities. However, in some DHS countries there is a growing demand for data from university students, indigenous NGOs and research companies. In this regard, it is not surprising that datasets from India are the most requested from the MEASURE DHS website. Gaps and unmet needs for capacity building In addition to the capacity building activities that have been undertaken, the evaluation team heard of additional needs or strategic approaches that could enhance national capacity in data use and survey implementation. First, the demand for report writing, data use and data analysis workshops is likely to continue. In-country stakeholders place a high value on increasing the capacity of individuals in government, NGOs and media to access, understand, interpret and apply DHS results. ICF has developed several useful tools, such as the DHS Curriculum and a handout describing how to read a DHS table. Investment in producing high quality EXTERNAL EVALUATION OF MEASURE DHS 22 presentations and supportive learning materials to help people who are not accustomed to deriving meaning from numbers would be highly valued. Like all DHS material, once finalized these learning materials could be made available on the web to encourage wider use. Second, but closely related to training materials, is the presentation of the results themselves. ICF’s communication advisors were praised for their knowledge and receptiveness to new ideas, but the unit is too small for the demands coming from countries. Every stakeholder has its own needs. Agencies involved in advocacy wanted a human face to the numbers and service implementers want to translate regional or provincial findings to the district. In particular, there is a growing demand for more visual displays of information, including the use of video and interactive data presentations. The medium of conveying DHS information has not kept up with this new technology. The GIS displays soon to be available on the MEASUREDHS.org website will help to meet some of these needs. The third gap is in building national capacity in advanced demographic and health data analysis. The key in￾country partners for DHS are the LIA (the national statistical office or a survey research firm) and the Ministry of Health. However, faculty and staff of universities and other autonomous or semi-autonomous research institutes are also potential beneficiaries of efforts to build further analysis capacity. In addition, such organizations are a reservoir of skills in training and analysis that could be used by government and NGOs to make greater use of DHS data. Building capacity, however, without defining future roles for said entities may make the efficiency or effectiveness of such investments very difficult to determine. Barriers to capacity building Stakeholders recognized many barriers to capacity building, and the evaluation team was able to observe them in operation during the site visits. The LIA’s and Ministries of Health often have high turnover of young professionals with technical skills. Longer-term staff may not have the incentive to master new skills. This presents a problem for individual-based capacity building strategies such as on-the-job training and workshops because the effort does not necessarily result in sustained capacity development. Equally frustrating is that when opportunities for capacity building present themselves, there are often not the funds to pay for it. Capacity building activities usually can only be considered when other survey implementation costs are already fully budgeted so, in practice, only USAID Missions with a large health program are likely to support capacity building activities. The contractor usually waits for the Mission to take the lead in suggesting additional capacity building efforts. Equally important is the lack of systematic planning for capacity building. Our informants told us that assessing institutional or individual capacity and development goals was not part of the process of preparing for a DHS. The country plans, an initiative required by the RFP for MEASURE III DHS to place the DHS into a broader health data collection context, does not include a capacity assessment or describe other data￾related capacity building activities by the government or donors. Weaknesses of the current approach The MEASURE DHS Capacity Building Strategy approved by USAID has three inherent weaknesses. The first is that it is dependent on additional funded activities such as workshops that are often not part of original project budgets. USAID/Washington funds regional workshops and training conducted in the United States. The USAID Missions and other donors must fund the intensive in-country work with national institutions. In practice this means that countries with little funding and high needs for strengthening local capacity are the least likely to receive capacity building efforts. On-the-job training and opportunistic training during data processing visits and report writing or data usage workshops will be the extent of capacity building efforts in these contexts. EXTERNAL EVALUATION OF MEASURE DHS 23 The second weakness is that it operates without a clear set of activities linked to measurable objectives at the country level or globally. Because additional funds are not allocated the needs assessment activities tend to be informal. The full package of TA does not include a formal assessment of learning needs or individual and institutional work plans with measurable targets. The third weakness is that capacity building is conceived of as the transferring of only technical skills. If institutions are to be targeted as well as individuals, finances, HR, management issues take on greater relevance as well. ICF may need to work with organizational development firms or organizational partners to determine what the needs are with key stakeholders and thereafter facilitate changes that affect DHS implementation. Many in-country respondents mentioned the financial challenges faced by the NSO’s and other partners. 6. Conclusions The data presented in the Main Findings section of this report were based on detailed interviews with more than 140 respondents representing the full range of DHS stakeholders, e-survey results Missions in developing countries, and a review of a large number of reports and other documents provided to the evaluation team by ICF. During our analysis of the interviews and e-survey data, we saw common statements expressed by many respondents within and across stakeholders groups. When facts, views, or concerns were important, valid, and often expressed, they were used to form our main conclusions. In determining what conclusions to draw from our results, views from all the various stakeholder groups were taken fully into account and if different, discussed. Our main conclusions are presented below, organized by the main themes of the evaluation. A. Meeting Stakeholders’ Needs 1. DHS is a highly valued data source that is used extensively for policy-making, strategic planning, program monitoring and management, and monitoring of international indicators. 2. In case study countries, there was widespread interest in developing less technical thematic policy briefs that would distill and discuss the policy/program implications of DHS findings. ICF could do more to encourage that these sorts of documents be written, printed and disseminated in￾country. 3. DHS Phase III has sponsored a national dissemination seminar in every country surveyed; and it has organized additional Dissemination Workshops at the sub-national level in some countries. Workshops on Understanding/Using DHS Data have been conducted in about 10 countries, but not all; and the same is true for Further Analysis workshops. 4. There still is substantial untapped potential for additional in-country utilization and analysis of DHS data. This potential represents unrealized return on the substantial investment already made in collecting DHS data. 5. Re-doubling efforts to promote further utilization and analysis of the DHS data, leveraging other partners in-country to assist in the effort, could be a relatively low cost strategy, with a huge potential benefit. 6. There is always a tension between efficiency and the full involvement of other stakeholders. Collaboration and coordination take time, involve negotiation and consensus building, and can involve managing competing demands. In implementing MEASURE DHS, ICF is under pressure to work efficiently and work closely with many other stakeholders, including a myriad of donors, international organizations, US government agencies, and in-country government ministries, NGOs and universities. Therefore, the challenge is finding the proper balance. EXTERNAL EVALUATION OF MEASURE DHS 24 7. ICF generally seems to appreciate the need for and benefits of stakeholder involvement, and it is doing a lot of things right. For example: • With USAID support, ICF has actively encouraged other donors to provide funding for DHS surveys, and donors have responded. • In virtually every country where DHS has been conducted, a National Steering Committee, comprised of representatives of dozens of key stakeholders, has been established to manage the design and implementation of the survey. • During the survey design stage of DHS surveys, ICF generally has encouraged country stakeholders to propose additional questions. In addition, a local Technical Working Group, acting under the authority of the National Steering Committee, has made final decisions on questionnaire content. • ICF consistently has respected the data rights of participating countries by guarding against any unauthorized release or use of survey data prior to the publication of the final report in￾country. 8. There are still areas, however, where ICF could do a better job of involving local stakeholders in survey planning and implementation. For example: • Some implementing agencies have reported that they have not always been properly involved at a key stage of survey implementation. • In some case study countries, MOH officials commented that the MOH should have been more involved, especially in data dissemination, utilization, and Further Analysis workshops. 9. In countries where host country technical experts are involved in writing chapters of the final report, we received very positive feedback about this experience. However, where this was not done, we heard feedback that local experts should have been more involved. B. Effectiveness of Technical Assistance 1. The number and types of ICF TA visits provided sometimes are determined mainly by ICF, without full discussion with the implementing agency regarding their capabilities and technical assistance requirements. 2. The predominant TA model used by DHS is short-term visits by ICF staff to assist LIA staff at each of the key stages of survey implementation, which we will call the OJT model. Stakeholders we interviewed overwhelmingly gave high marks to ICF staff for their skill and professionalism, and for their effectiveness in helping to implement high quality surveys consistently. 3. Inclusionary TA and capacity building approaches has awakened a heightened sense of momentum and desire for responsibility and skills amongst LIA’s. ICF and USAID/W will need to be cautious of heightened expectations and illustrate a transparent process whereby TA is monitored against operational goals and targets and readied institutions and individuals appropriately manage aspects of survey implementation. 4. ICF also utilized other models for TA, including resident advisors (Malawi, Bangladesh) and locally hired expert consultants (Malawi, Bangladesh, and Cambodia). Based on data presented in our case study countries and on the e-survey, stakeholders were impressed with the quality of work by these consultants, and they appreciated working with staff from their own region, which are familiar with their culture and (in some cases) also speak their native language. Another advantage raised by stakeholders of using a resident advisor is that s/he is always available for consultation and mentoring, which helps to accelerate the transfer of skills. EXTERNAL EVALUATION OF MEASURE DHS 25 5. Many stakeholders in case study countries and in USAID also supported the general idea of regional DHS offices staffed with full time advisors who could take some of the TA burden off the DHS HQ staff. However, it is possible that greater use of south-to-south TA (see Conclusion B.4 above) would be a more cost-effective TA model. C. Task 2: Effectiveness of Capacity Building Activities 1. Over the years, the DHS has had a positive impact on survey capacity building in many countries regardless of a budget or plan for such activities. Examples of success include: • Philippines and Indonesia are successfully managing their own DHS surveys without TA from ICF. • Colombia is also implementing its own surveys, and is in fact providing south-to-south TA to other countries in the region • Tanzania, Uganda, Nepal and Bangladesh have made significant gains in survey capacity. 2. Efforts to strengthen capacity also face tough challenges, including: • Survey content is ever-changing, and becoming more complex • Surveys normally are done only every 3-5 years • Trained, experienced people leave and go elsewhere • DHS and USAID mission funding for capacity building has been limited 3. Technical assistance visits may help to build capacity, but only if they involve the full engagement and participation of host country counterpart staff. Because country level partners have been asked to get involved in the implementation of the surveys, there is increasing pressure to make every step in the process open to local partners and transparent. 4. Further, most respondents cited utilization and analysis of the data as a high priority for capacity building. There is also considerable interest in becoming better experienced in the use CSPro, so that host country staff can do more of the data processing work themselves. We note also that ICF is developing tabulations for all chapters in the final report in SPSS and in some cases also in STATA, which should make them more accessible to analysts in the country. 5. Capacity building within current programs does not match the demand, nor does it anticipate well the needs by country. To reach a level of cost-effectiveness it must identify desired goals operational milestones, CB requirements by institution and country, and monitor the efficacy of CB work overtime. 6. Graduation from DHS assistance is both a technical and political issue. A transition plan is needed to ensure an orderly graduation process, and allow sufficient time to find the necessary survey funding in the country’s national budget. 7. Recommendations A. SUMMARY KEY RECOMMENDATIONS What is working well? EXTERNAL EVALUATION OF MEASURE DHS 26 1. As noted previously, the DHS continues to provide high-quality timely data project-wide. The engagement of stakeholders at international and country-levels through Committees has been successful and should continue. Further, the excitement and energy around DHS implementation is evident and the desire for countries to take on more or in some cases all of the survey functions based on largely on-the-job capacity building is impressive. It illustrates an appreciation for the work of the ICF team and a desire to better use ICF time and resources. 2. While all stakeholders see a continued and important role for ICF the work, particularly in recent DHS cycle has ignited a growing demand for increased responsibility and ownership of every stage of the survey process. A future vision for DHS should be articulated and planned for to ensure quality, comparability and access are maintained; and, the increased excitement around country ownership and country-led processes met with a shared vision of the future DHS program. 3. Technical assistance will continue to be the main strategy for this and a substantial cost. Localizing TA inputs through consultants, technical advisors and centers of excellence could reduce cost over the long run but also increase costs in the short run. A combination of approaches will likely be required, and a specific scope of work for each implementer, including ICF required should the approaches be measured against the stated goal/outcome of increased implementation capacity. What could be improved or needs changing? 1. Regarding the growing size of the DHS along with the demands for smaller geographic estimates, the team proposes that USAID become much more involved in the support of a broader approach to health data. There is a need to look broadly at the role of household surveys with the aim of complementing surveys with censuses, surveillance systems, and HMIS. This is beyond the scope of MEASURE III DHS to manage, but should be considered, debated and costed for future USAID and other donor programs. Additionally, clearer technical governance processes should be considered to assist in containing the length of the survey from questions that could be obtained in other ways. 2. The DHS will remain heavily dependent on TA in the future. A short-term approach to TA concerns would have to address the number and purpose of TA visits and discussed amongst all stakeholders at the design phase. Local experts could be identified and a job description developed that outlines how they will work with ICF advisors. The evaluators appreciate that some of this information may be commercially sensitive. 3. In terms of capacity building approaches into the future, much of this would be informed through evaluation of factors in some countries that no longer rely on MEASURE DHS TA, including enablers and barriers to producing quality data for their own programming needs and contributing to international collections of comparable, high quality data. The evaluation team recommends that funding mechanisms for survey implementation and sustainable capacity should also be investigated. 4. Capacity building should be measured against global targets and milestones in implementing capacity reached by country partners. Explicit capacity building goals and objectives (individual and institutional) would also be weaved into country-level five-year work plans and strategies. EXTERNAL EVALUATION OF MEASURE DHS 27 B. DETAILED RECOMMENDATIONS/DISCUSSION BY TASK A. Meeting Stakeholder Needs 1. Regarding data dissemination, the evaluation team proposes that in each country, a plan for data dissemination, utilization and further analysis should be developed in collaboration with the USAID Mission and the National Steering Committee. The implementation of the plan could extend 1-2 years after the publication of the final report. Discussion: Our recommendation calls on ICF to make a more systematic effort to promote greater dissemination, data utilization, and further analysis in every country undertaking DHS surveys by providing options to countries. Approaches that are popular could become part of the standard output packages – e.g. a health handbook for teachers and journalists or a women’s health fact book – while others might just inspire countries. Then the stakeholders pick from the menu, add their wishes and the consortium in the country finds the funding. In addition to the obvious in-country costs associated with this recommendation, it is possible that there may be an increase in HQ staff efforts to help negotiate and implement a larger program of workshops worldwide. However, as noted in paragraph above, the team sees potential for ICF to leverage in-country partners (e.g., MOH staff; or NGOs) to partner in organizing/leading some workshops in the plan (especially Dissemination or Understanding/Utilizing DHS Data workshops) using a TOT approach. Costs for using these partners would likely be less than the in-country and HQ costs for ICF doing all the work alone. 2. ICF should make greater efforts to distribute DHS findings to district level health program staff. There is substantial demand for this, but not enough resources and time to address this well. The evaluation team proposes that MEASURE DHS develops guidance on best practice for using the data for district information and then let countries figure out how they choose to proceed. Discussion: Some stakeholders recommended ICF produce less technical program-oriented Fact Sheets and/or policy briefs to be made available at the district level, because some districts do not have access to copies of the DHS final report. Having ICF be the conduit to paying for the materials would be an opportunity for them to review them for accuracy but that might discourage some use. This is why the evaluation team proposes a menu – every country gets instructions on interpreting the sub-national data, a subset get some TOT training and another subset get review and printing of materials. For instance, in Cambodia, the fast facts type of poster had district level indicators on the back, but without standard errors. Costs for developing these materials would not be large, but the benefits of providing useful and accessible data to health workers on the “front line” of health service delivery would be substantial. 3. ICF should encourage the development of less technical thematic policy briefs that would distill and discuss the policy/program implications of DHS findings. In most cases, these policy briefs could be written by local experts, and printed and disseminated in-country. Discussion: In general, the team noted that DHS reports and publications by design are not oriented toward discussion of policy or program implications of the data. However, we saw evidence of a clear unmet need among in-country MOH leaders, NGOs, journalists, and donors for just this type of less technical thematic “policy briefs” publication. This recommendation does not ask ICF to draft or produce such publications, but simply to encourage their production and dissemination by institutions with a ‘research environment’. Organizations with this capacity vary widely between countries. In general, the institution should have some of its own infrastructure and not depend EXTERNAL EVALUATION OF MEASURE DHS 28 solely on fee-for-service. These policy briefs would be fairly low cost to produce, but would be well￾received by in-country stakeholders. 4. Of those who wanted a “fresher look” to the DHS, they asked that DHS reports strike a better balance between the reporting of essential outcome and impact indicators (with standardized comparability across countries) and in-depth treatment of special topics identified by host countries. Minimally, the evaluators recommend that ICF should be sufficiently flexible in the content of its final reports to allow for tabulation and analysis of any/all special questions added to the questionnaire at the request of local stakeholders. Discussion: The team recognizes the advantages of a standardized format for writing DHS final reports. However, DHS encourages in-country stakeholders to submit survey questions for a process of technical vetting and approval by the National Steering Committee. Naturally, questions that are approved for inclusion in questionnaires ought to also be included in the final report tabulations and analysis. Perhaps in-country authors could even be asked to draft the analysis on these special questions. 5. To the extent possible, the implementation of every stage of a DHS survey (including sampling, data processing and data analysis and report writing) should be done in-country, with the full participation of host country staff. If the need arises to complete some task(s) at ICF HQ, that decision should be made jointly by ICF, the LIA, and the National Steering Committee; and every effort should be made to bring host country staff to the US to participate in that work. This approach is crucial to building survey capacity, fostering goodwill with host country implementing partners, and creating a full sense of country ownership of the DHS survey. Discussion: The team recognizes that in general it is ICF’s policy to involve the LIA in all stages of DHS surveys. However, the team heard from at least two country case study implementing agencies this was not always done, especially in some of the more technically demanding survey operations (e.g., sampling, data processing, and analysis/report writing). Therefore, the above recommendation is intended to reinforce the importance of full country participation in the survey, and the need for balancing the priority to “get the survey done fast” with equally important priorities related to country participation and ownership, and capacity building. The costs of implementing this recommendation include slightly longer TA visits in some cases to allow more time for mentoring/teaching, and perhaps some delays in completion of surveys. Time delays are a major concern of local and international-level stakeholders. But greater hands-on involvement by LIA staff in all stages of the survey will accelerating capacity building and strengthen their sense of country ownership of the survey. 6. Special efforts should be made to encourage greater involvement of the MOH in DHS surveys generally, and especially as partners for facilitating dissemination, utilization, and further analysis of DHS data for management and improvement of health policies and programs. Discussion: The team found that the MOH is a very important host government user of DHS data, with a high interest in increasing DHS data utilization within the Ministry and among its NGO partners. Therefore, key technical staff in all relevant departments of the MOH should have increased access to Dissemination Workshops, Understanding/Using DHS Data workshops, and Further Analysis workshops. As noted earlier, ICF should enlist the MOH as a TOT partner to help ICF roll out these workshops to MOH staff and partners outside of the capital city, as needed. B. Effectiveness of Technical Assistance 1. TA requirements for each DHS survey should be worked out during the survey design phase in discussion with the LIA, taking full account of their existing technical capacity, and their prior experience with DHS. EXTERNAL EVALUATION OF MEASURE DHS 29 Discussion: Based on discussions with ICF and USAID/W staff, and interviews with implementing agencies in case study countries, the team found that the “full package” of 10-12 ICF TA visits is the default model for TA, employed in about 70% of DHS surveys (according to e-Survey results). We also learned that ICF visits are not just about providing TA; they are also necessarily used as a tool for monitoring survey progress and “keeping things moving.” This may explain why we found that often there is not a full discussion of TA requirements with the LIA during the survey design stage so that TA plans can be tailored to the LIAs specific technical capabilities and needs. Our recommendation above affirms that TA plans should be fully discussed with LIAs and tailored to their technical capabilities and needs. This would be addressed should the recommendation that all countries be on a continuum towards country-led DHS process, and each participate in a planning process which includes assessment of capacity, 5-year goals, objectives and operational milestones, etc. The transparency of this kind of process may help manage expectations in country and build greater understanding that TA has other functional purposes in the short run in terms of moving the process along, but should have built-in triggers for changes in the modus operandi. 2. In view of the positive feedback received on regional resident advisors and part-time expert consultants, ICF should consider more widespread use of these models of TA, with the goal of relieving some of the TA burden on ICF HQ staff. For instance, future MEASURE DHS contracts could stimulate the provision of south-to-south TA by requiring US contractors to have regional partners responsible for providing local experts for this purpose. Efficiencies and effectiveness measures will need to be developed as part of a ‘transition’ plan to local implementing partners with operational milestones detailed. Training provision without and endpoint in sight, though effective at generation “good will”, will likely proof wasteful and duplicative of work having to be done by the main contractor each DHS/MIS/AIS cycle. Discussion: Making greater use of high-performing regional staff as resident advisors or consultants represents a lower cost alternative (in terms of salaries and travel costs) to sending ICF staff, if these visits could substitute for ICF visits. 3. ICF also should explore possibility of opening at least one regional office (e.g., in Africa), based on a cost-benefit comparison of this model with other available TA models. Discussion: In a region like Africa, where there is a consistent high level of ongoing DHS survey work, opening a small DHS regional office could speed the deployment of TA and it certainly would result in lower travel costs to the region. However, some of these savings may be offset by allowances that may need to be paid to expatriate staff in a regional office. Another alternative would be to identify and contract with centers of excellence that could serve as local sources of TA expertise (after proper orientation and training by ICF) in-country. 4. ICF should consider making greater use of video or webinar technology for certain types of training/TA purposes, possibly in lieu of some TA visits. If feasible, use of this technology could prove very cost-effective. Discussion: International organizations and USAID contractors increasingly are making use of video conferencing and webinars for training purposes. Although bandwidth issues are still a problem in rural areas of some of the case study countries we visited, Internet connectivity in most capital cities is sufficient to support use of these technologies; and Internet access is continually improving. Furthermore, these technologies have most of the advantages of on-site training, but at a lower cost; and a training workshop can be accessed in several countries simultaneously, if needed, generating significant economies of scale. We understand that ICF has already begun exploring some options for use of technology as an alternative to in-person visits for TA and training purposes. EXTERNAL EVALUATION OF MEASURE DHS 30 TA is a major service provided by ICF under the MEASURE DHS contract and should be part of regular reporting to USAID. In addition to documenting outputs, ICF should reflect, document and analyze progress against institutional and country level operational targets. Discussion: USAID ought to be able to monitor trends in the volume and costs trends in TA provision over time, and compare the costs of different TA models that are employed. This will keep USAID better informed on DHS TA activities, and assist efforts to optimize TA efficiency and effectiveness. C. Effectiveness of Capacity Building Activities 1. Plans for supporting further dissemination, utilization, and analysis of DHS data in each country should be implemented in ways that will also contribute to building of host country capacity. Discussion: The team sees a great deal of synergy between efforts we are recommending to increase data dissemination, utilization and further analysis and efforts to build capacity. Indeed, improving skills for increased data utilization and further analysis were among the top priorities for capacity building mentioned by case study stakeholders and e-Survey respondents. By making this synergy intentional, the investments in improving data utilization and further analysis can effectively work to achieve two key objectives of MEASURE DHS. 2. Planning for capacity building activities should occur during the survey design phase of surveys, so there can be dialogue on capacity building goals, priorities, and benchmarks; and so the costs for capacity building activities can be included in the survey budget. Discussion: For the reasons mentioned above, the team suggests that the country planning exercise recommended here for capacity building, and also recommended earlier for improving data dissemination, utilization and further analysis, should either be integrated or closely coordinated. These planning exercises are closely related, and both are intended to occur during the survey design phase. 3. The concept of graduation of countries from DHS is not well defined based on technical or other factors. Graduation was not a major focus of this Scope of Work, however capacity building and TA effectiveness must be planned against a specific outcome or vision for future DHS. Discussion: Achieving full technical competence for conducting high quality household surveys could be one goal of the DHS program, or achieving full technical competence to conduct a DHS survey another. Because this question is beyond the scope of the evaluators knowledge and questions, we recommend that this issue be discussed and clarified amongst USAID/W in order to plan outcomes-focused M and E and costing around capacity building and TA activities. EXTERNAL EVALUATION OF MEASURE DHS 31 EXTERNAL EVALUATION OF MEASURE DHS 32 APPENDIXES A. Evaluation SOW B. List of USAID Mission Respondents Reached via E-survey C. PMI Contacts Reached via E-survey D. List of Documents Reviewed E. List of Key Informants Interviewed, by Organization F. Data Collection Instruments Developed F.1.Question line for Macro management and senior leadership F.2 Question line for Missions and Other Stakeholders in Country F.3 E-Survey Questionnaire G. Country Selection Criteria H. Country Case Descriptions I. Tabulations from the E-survey J. PowerPoint Presentation on the Results of the Evaluation EXTERNAL EVALUATION OF MEASURE DHS 33 A. Statement of Work Statement of WORK for an EXTERNAL EVALUATION of the MEASURE PHASE III DEMOGRAPHIC AND HEALTH SURVEY (DHS) CONTRACT I. Project to be Evaluated Project Name: MEASURE Phase III Demographic and Health Surveys (MEASURE DHS) Contract Contractor: ICF Macro Contract Number: GPO-C-00-08-00008-00 Contract Value: $142,462,784 Obligation Date: 30 September 2008— 29 September 2013 Funding Sources (evaluation): GH PRH/OHA/HIDN, cost share II. Purpose The purpose of this evaluation is to provide the United States Agency for International Development’s (USAID) Bureau for Global Health (GH) with an independent evaluation of key components of USAID’s flagship global health data collection project, MEASURE Phase III Demographic and Health Surveys project (hereafter MEASURE DHS). MEASURE DHS is a global project that seeks to improve the collection, analysis and presentation of data and promote better use in planning, policymaking, managing, monitoring and evaluating population, health and nutrition programs. The project seeks to increase understanding of a wide range of health issues by improving the quality and availability of data on health status and services and enhancing the ability of local organizations to collect, analyze, disseminate, and use such information. MEASURE DHS is operating in a complex and rapidly changing development landscape, where there is increased complexity associated with negotiating content and managing the implementation of DHS surveys. This includes continuing emphasis on aid effectiveness principles, the expanded role of USG partners within the GHI, country-specific context and capacity, and sometimes conflicting global-level and country-level data needs. In addition, the MEASURE DHS contract lays out a vision for building capacity of local institutions and individuals along the data demand-collection-use continuum, while at the same time ensuring high quality products in partnership with countries with highly variable levels of capacity in survey implementation and statistical analysis The key tasks (and questions that will guide) of this evaluation are to: 1. Provide a cost/benefits analysis of how MEASURE DHS operates within this complex and changing environment, with an emphasis on the following: a. How successfully the contractor is responding to the varied needs of key stakeholders, and which needs are within the manageable interest of the Project. b. How successfully the contractor is providing technical assistance in survey content, design, and implementation. What technical assistance approaches (including those that may not be part of the current Project approach) might increase the contractor’s performance in this area? 2. Provide an analysis of the feasibility/appropriateness of the capacity building goals laid out in the contract in light of the increased complexity of survey work and the global demand for high quality survey data, with an emphasis on the following: EXTERNAL EVALUATION OF MEASURE DHS 34 a. What are the priority survey design/management and data analysis/use capacity building needs of key stakeholders? Are the capacity building strategies and activities undertaken by the Project meeting these needs, in particular the needs of Local Implementing Agencies (LIAs)? If not, what are the key barriers at different levels (i.e. design of the contract, stakeholder demand, funding, local “ownership”)? b. What capacity building approaches (including those that may not be part of the current Project approach) might increase the contractor’s performance in the area of building the capacity of local institutions and individuals in survey design/management and data analysis/use? 3. Provide recommendations for future directions in health data collection and survey work (based on Tasks 1 and 2). This task will involve documenting the following: 1) what is working in Tasks 1 and 2 that should be maintained; 2) what is working in Tasks 1 and 2, but needs to be improved because of inefficiencies or ineffectiveness of practices; 3) what is not working in Tasks 1 and 2 and needs to be changed. The evaluation will gather and synthesize information from multiple sources, including the GH Bureau Offices (Population and Reproductive Health, Office of HIV/AIDS, and Health, Infectious Disease and Nutrition), Regional Bureaus, Missions, ICF Macro and its partners on the MEASURE DHS Project, and other stakeholders and subject matter experts. Key findings and recommendations will serve to provide an overview of and promising practices in survey implementation and health information capacity building activities. It should be noted that this evaluation will not have as a major technical focus documenting the project’s overall achievements toward targeted results. Because DHS is a cost-plus award fee contract the performance of the contract is monitored annually through a rigorous process of establishing and measuring progress towards performance benchmarks. Thus, the USAID management team determined that it would be redundant to include this as a major technical focus of the external evaluation. III. Project Background The DHS Project (also known as MEASURE DHS) has conducted more than 240 surveys in about 85 countries since its inception in 1984. Of these surveys, nearly 50% have been conducted in 38 Sub-Saharan African countries and the rest have been conducted mainly in Asia and Latin America. Over the past five years, an increasing number of countries in Central Asia and Eastern Europe have also participated in the MEASURE DHS program. As is the practice in other regions, the project has had to adapt its survey instruments to respond to these countries’ specific data needs. The core operations of the MEASURE DHS project is designed, managed, and funded by the U.S. Agency for International Development (USAID). Contributions from other donors, as well as funds from participating countries, also support surveys. The project is implemented by ICF Macro, an ICF International Company. Since October 2008, ICF Macro has partnered with five internationally experienced organizations to expand access to and use of the DHS data: Johns Hopkins Bloomberg School of Public Health/Center for Communication Programs ,PATH, The Futures Institute, CAMRIS International, and Blue Raster. Strategic Objective and Results The overall objective of MEASURE DHS is improved collection, analysis and presentation of data to promote better use in planning, policymaking, managing, monitoring and evaluating population, health and nutrition programs. This overall objective is to be accomplished through the achievement of six intermediate results: EXTERNAL EVALUATION OF MEASURE DHS 35 • Result 1: Increased user demand for quality information, methods, and tools for decision making; • Result 2: Increased in-country individual and institutional technical/managerial capacity and resources for the identification of data needs and the collection, analysis and communication of appropriate information to meet those needs; • Result 3: Increased collaboration and coordination in efforts to obtain and communicate health, population and nutrition data in areas of mutual interest; • Result 4: Improved design and implementation of the information gathering process including tools, methodologies and technical guidance to meet users’ needs; • Result 5: Increased availability of population, health and nutrition data, analyses, methods and tools; and • Result 6: Increased facilitation of use of health, population and nutrition data. In pursuant of these six intermediate results, MEASURE DHS provides technical resources and assistance to host-country partners, USAID missions, and cooperating agencies as they develop and implement plans to increase demand for data; define information needs; and collect, translate, disseminate, analyze and use demographic and health data. MEASURE DHS also contributes to in-country sustainability of optimal population and health program planning, management, and policy development by building local capacity in all aspects of data collection, monitoring and evaluation, and further analysis of data. According to USAID expectations, efforts to optimize, increase, and sustain host country capacity and ownership guides all technical assistance and implementation activities undertaken by MEASURE DHS. MEASURE DHS implements an array of data collection approaches, including but not limited to the following: • Standard Demographic and Health Surveys (DHS) – These are nationally representative household surveys conducted every five years with large sample sizes to allow comparisons over time. These surveys provide data for a wide range of monitoring and evaluation indicators in the areas of population, health and nutrition (PHN). The core questionnaires—household-, female individual-, and male individual-questionnaires—form the basis for the questionnaires that are applied in each country. • Interim Demographic Health Surveys (Interim DHS) - These smaller surveys are conducted between rounds of the standard DHS to provide information on key performance monitoring indicators. • AIDS Indicator Survey (AIS) – The AIS was designed to provide countries with a standard survey protocol to obtain information required for meeting HIV/AIDS program reporting requirements in a timely fashion and at a reasonable cost. The reporting requirements include the collection of indicators, including those of the President’s Emergency Plan for AIDS Relief (PEPFAR) and United Nations General Assembly Special Session on HIV/AIDS (UNGASS), and ensuring comparability of findings across countries and over time. The AIS consists of two survey instruments – the household questionnaire and the individual questionnaire. The AIS is sometimes integrated into the standard DHS. • Malaria Indicator Survey (MIS) – The MIS was developed by the Roll Back Malaria Monitoring and Evaluation Reference Group (RBM MERG) Survey and Indicator Task Force, with MEASURE Phase II DHS highly involved in its development. The MIS was designed as a nationally representative, household-level survey package for assessing coverage of insecticide-treated mosquito EXTERNAL EVALUATION OF MEASURE DHS 36 nets (ITN) based on a full net roster, anti-malarial treatment among children under five with fever, and intermittent preventive therapy (IPT) among pregnant women. The MIS is sometimes integrated into the standard DHS. • Service Provision Assessment (SPA) – The SPA is conducted in health facilities and communities to obtain information about the characteristics of health services including their quality, infrastructure, utilization and accessibility. The SPA includes observations and provider interviews to assess the capacity of facilities to provide services that meet a given standard and maintain that standard over time; client exit interviews and observations are also used to assess service providers’ adherence to standards in the provision of care. • Collection of Biological Specimens (Biomarkers) and Measurements – Biomarkers are biologic measures of health conditions. In addition to collecting information on height and weight to assess nutritional status of women and children, MEASURE DHS uses field-friendly technologies to test for syphilis, the herpes simplex virus, serum retinol (vitamin A), malaria parasites, lead exposure, high blood pressure, and immunity from vaccine-preventable diseases like measles, and tetanus, while conducting Demographic and Health Surveys. Most surveys now include testing for HIV infection in their design. It should be noted that this evaluation will potentially cover all of the above activities/survey types. At the beginning of the MEASURE DHS, the sample size of each survey was typically less than 6,000 respondents. Over the years, as a result of increasing demands for sub-national (regional and district) level estimates, the sample size of most DHS-implemented national surveys has increased. In addition, the range and depth of topics have expanded. Initially, respondents were only women of reproductive age and issues addressed consisted mainly of fertility, family planning and maternal and child health. Currently, most of the surveys include male respondents and data are collected on a wide variety of topics that include anemia, malaria, maternal and child nutritional status, geographic location, HIV, syphilis, and vitamin A deficiency. In responding to these changing data needs, the MEASURE DHS project has been flexible without losing focus or sacrificing data quality. IV. Statement of Work As previously stated, MEASURE DHS operating in a complex and rapidly changing development landscape, where there is increased complexity associated with negotiating content and managing the implementation of DHS surveys. This includes continuing emphasis on aid effectiveness principles, the expanded role of USG partners within the GHI, country-specific context and capacity, and sometimes conflicting global-level and country-level data needs. In addition, the DHS contract lays out a vision for building capacity of local institutions and individuals along the data demand-collection-use continuum, while at the same time ensuring high quality products in partnership with countries with highly variable levels of capacity in survey implementation and statistical analysis. Within this changing context, there are three main tasks associated with this evaluation are to: • Task 1: Provide an analysis of the costs/benefits of how the DHS Project operates within this complex and changing environment, with an emphasis on the following: o How successfully the contractor is responding to the varied needs of key stakeholders, and which needs are within the manageable interest of the Project. o How successfully the contractor is providing technical assistance in survey content, design, and implementation. What technical assistance approaches (including those that may not be EXTERNAL EVALUATION OF MEASURE DHS 37 part of the current Project approach) might increase the contractor’s performance in this area? (Estimated level of effort – 40%) • Task 2: Provide an analysis of the feasibility/appropriateness of the capacity building goals laid out in the contract in light of the increased complexity of survey work and the global demand for high quality survey data, with an emphasis on the following: o What are the priority survey design/management and data analysis/use capacity building needs of key stakeholders? Are the capacity building strategies and activities undertaken by the Project meeting these needs, in particular the needs of Local Implementing Agencies (LIAs)? If not, what are the key barriers at different levels (i.e. design of the contract, stakeholder demand, funding, local “ownership”)? o What capacity building approaches (including those that may not be part of the current Project approach) might increase the contractor’s performance in the area of building the capacity of local institutions and individuals in survey design/management and data analysis/use? (Estimated level of effort – 40%) • Task 3: Provide recommendations for future directions in health data collection and survey (based on Tasks 1 and 2). (Estimated level of effort – 20%) Information should be gathered from GH Offices, Missions, Regional Bureaus, and additional stakeholders (to be determined prior to the evaluation), partners and subject matter experts. A recommended but not exclusive list of interviewees is provided in Annex 1. Listed individuals will need to be grouped by audience for input to Tasks 1- 3. V. Methodology Data Collection The evaluation team will work collaboratively with the USAID management team to develop a detailed work plan as well as a data collection strategy including data collection instruments. A variety of methods including analysis of information and data obtained through, but not limited to, document review, key informant interviews, surveys, site visits/direct observation, and desk review of relevant technical literature will be included. Data collection approaches are described in more detail below. Documents Review (project and technical): a. MEASURE DHS project framework b. MEASURE DHS RFP c. Relevant DHS Award Fee responses, Year 1, Year 2, Year 3 d. Capacity building strategy e. Concept note (and other relevant materials) for DHS short term fellows f. Concept note (and other relevant materials) for DHS junior faculty program g. Relevant country plans h. Relevant trip reports i. Past evaluation of the Peru Continuous Survey experiment j. GHI guidance document, including priority countries EXTERNAL EVALUATION OF MEASURE DHS 38 k. BEST technical guidance documents l. Desk review of survey sustainability and capacity building approaches in health data collection efforts m. CDC Report: Assessment of Factors Affecting National Population-based Health Surveys in Four Latin American Countries (2009) n. USAID Evaluation Policy Key Informant Interviews The team will conduct semi-structured interviews with key stakeholders, including but not limited to the following. A final list will be developed in conjunction with the evaluation team, who may also suggest additional persons who can provide insights useful to this evaluation. USAID interviews a. BGH (PRH, HIDN, OHA) staff, including those working/have worked on health data collection projects (Jacob Adetunji, Noah Bartlett, YJ Choi, Rachel Lucas, Lisa Maniscalco, Madeleine Short, Noni Hamilton, Misun Choi, Krista Stewart, John Novak, etc). b. Other key BGH stakeholders including but not limited to Scott Radloff, Ellen Starbird, Liz Schoenecker, Sarah Harbison, Mihira Karra, Richard Greene, Robert Clay, members of SPER/OHA, PMI team members, etc. c. Stakeholders in the Policy, Planning, and Learning (PPL) Unit, such as Ruth Levine. d. Regional bureau staff with an interest in DHS: Gary Cook, Jennifer Mason, Ishrat Husain, Lindsey Stewart. e. USAID/Mission staff that support DHS and other ICF Macro-sponsored surveys (AIS, MIS, etc) in select countries, TBD. DHS Project Staff interviews a. MEASURE DHS staff including key personnel, select country managers, in country DHS staff in select countries (if applicable0, communication/dissemination team, and others who work on capacity building activities. External Informants (will include but is not limited to) a. MEASURE partners (CDC, MEASURE Evaluation, BUCEN) b. In-country statistical authorities/offices in focus countries c. Host government stakeholders, as defined d. UNICEF e. IHME f. Universities, journalists, and other stakeholders who are focus of capacity building activities Key informant interviews with domestic stakeholders will be conducted in Washington, DC at their respective on-site locations or by telephone, whichever is most expedient and cost effective. Additional telephone interviews with overseas interviewees with Mission staff and other external stakeholders may be conducted from the U.S. Others will be added to the lists by the team as additional key informants are identified during the course of the evaluation. EXTERNAL EVALUATION OF MEASURE DHS 39 Site Visits USAID envisions that a select number of countries (2-4) with active survey schedules that are well positioned to provide diverse perspectives on the evaluation questions (for example: Kenya, Tanzania, Malawi, Honduras, Jordan, Ghana, Senegal, etc) would be selected as case studies for the two technical areas of focus (Tasks 1 and Task 2). The evaluation team will consult with and receive approval from the USAID management team as to the selection of countries for case studies. VI. Team Composition A three-member evaluation team is proposed; one person will be designated as the team leader and will be in charge of the overall design, data collection, analysis and writing of the evaluation report. The three person team will be comprised of the following: o Team Leader and Survey Specialist will oversee all aspects of the project, liaise with the other consultant, and with USAID/BGH, oversee data collection and analysis, write sections of the report, meld contributions of the Technical Consultant into a coherent set of responses and present conclusions and recommendations to USAID. The team leader should have prior experience and expertise in program evaluation and assessment, survey implementation expertise, understanding of USAID program and processes, and experience with international population and health technical areas. o Technical Consultant who has specialized evaluation expertise and programmatic experience in capacity building and/or organizational development programs, especially as it relates to capacity building of health information systems in developing countries. The technical consultant will bring the lens of his/her subject matter expertise and experience to bear on all aspects of the Scope of Work, with a focus on internal organizational development/capacity building aspects of this SOW. S/he will work closely with the Team Leader to assess the quality and relevance of internal work practices and processes, and offer his/her perspectives on tasks associated with this assignment. S/he will work seamlessly with the Team Leader to interview key informants, conduct data collection and analysis, and write sections of the report. o Technical Consultant who has specialized evaluation expertise and experience in survey operations, and will focus on external organizational development/capacity building aspects of this SOW. S/he will work closely with the Team Leader to assess the quality and relevance of capacity building activities, and offer his/her perspectives on tasks associated with this assignment. S/he will work seamlessly with the Team Leader to interview key informants, conduct data collection and analysis, and write sections of the report. The combined skill sets of the consultants should include, at minimum: • Longstanding experience and technical expertise in survey operations, especially in a developing country contexts; • Expertise in conducting evaluations and assessments of public health and/or capacity building interventions in developing country contexts; • Longstanding experience and technical expertise in population and health issues; • Excellent analytic and writing skills; EXTERNAL EVALUATION OF MEASURE DHS 40 • Familiarity with USAID systems and way of business (at least 1 consultant), and development context. VII. Duration, Timing, and Schedule USAID/GH anticipates that the period performance period of this evaluation will be approximately 91 days to begin on or about January 2012 (the actual start date will depend on the availability of consultants). Travel to the field will be required (these terms will have to be further defined and developed). Team Planning Meeting: The full team will have a one day planning meeting in Washington, DC. The team planning meeting is an essential step in organizing the team’s efforts. During this meeting, the team will meet with USAID DHS management team members to review the SOW and discuss expectations and deliverables, determine roles and responsibilities of all team members, and agree on a timeline for the evaluation effort. In addition, the following will be accomplished: Task/Deliverable Time in Days LOE Team Leader Team Member Team Member Total LOE (days) 1. Document review 2 2 2 2 6 2. Team planning meeting/Draft work plan/ scheduling of interviews 5 5 5 5 15 3. USAID reviews draft instruments /schedules 5 5 5 5 15 4. Team revises data collection instruments 2 2 2 2 6 5. Key informant Interviews and telephone interviews in Washington, DC (with USAID/W, CAs, Missions, experts, etc) 7 7 7 7 21 6. Site visits-2 -4 Countries (total) 12 12 12 12 36 Data analysis from Washington, DC (can als wrap up key informant interviews) 6 6 6 6 18 7. Mid-course meeting with USAID DHS management team to 1 1 1 1 3 EXTERNAL EVALUATION OF MEASURE DHS 41 • Clarification of any aspect of the SOW and what is expected • Logistical and administrative procedures for assignment, including field work • Agreement of components of the draft work plan • Establishment of a team atmosphere, through sharing of individual working styles, and agreement on procedures for resolving differences of opinion • Development of a preliminary draft outline of the team’s report, and • Assignments made regarding drafting responsibilities for the final report. Within three days following the Team Planning Meeting, the evaluation team will develop and submit to the USAID DHS management team a draft work plan that will include the following elements: • Description of each team members’ roles and responsibilities • List of final evaluation questions and/or guidelines for questionnaires • Approach to data collection, methodologies to be used, how data will be analyzed • Data collection instruments (to be included in appendices) • Draft outline of final report (as an appendix) • Assignment timeline The USAID DHS management team must approve the final work plan before data collection begins. discuss relevant issues, comments 8. Draft evaluation report; preparation for debrief 5 5 4 4 13 9. Debrief with USAID GH 2 1 1 1 3 10. Incorporation of debrief comments into draft report 1 1 1 1 3 11. USAID provides comments on draft report 10 0 0 0 0 12. Prepare final Evaluation report 3 3 1 1 5 Total number of business days 61 50 47 47 144 EXTERNAL EVALUATION OF MEASURE DHS 42 The USAID DHS management team and the evaluation team may request to meet on a periodic basis during the implementation of the evaluation for the purposes of clarification and sharing information. The following is a sample schedule. The evaluation team will finalize a schedule and exact dates for the evaluation at the Team Planning Meeting in collaboration with the USAID DHS management team. Proposed LOE is 91 person days distributed as follows: VIII. Deliverables The evaluation team will provide the following deliverables: Work Plan Within three days of the Team Planning Meeting, the team will submit a draft work plan to the USAID DHS management team via the contractor for approval. Any major changes to the work plan proposed by the evaluation team will be discussed with USAID and require approval by USAID prior to implementation. The draft work plan will contain the components listed above in Section VII. Debriefing After the evaluation team conducts all data collection and has time to discuss and reach consensus on the preliminary conclusions and recommendations, but before the draft report is submitted, the evaluation team will meet with the USAID DHS management team to discuss preliminary results. At this time, the USAID DHS management team may provide additional information and provide its perspectives, but will not approve the findings per se. A second briefing for a larger GH audience will take place several days after the draft report has been submitted to the USAID DHS management team. The evaluation team will prepare a PowerPoint presentation for the debriefing. Participants in the second briefing audience may include USAID/GH staff, and other interested persons outside of GH (USAID regional bureaus, PPL, etc). Draft Report The evaluation team will submit a draft report to USAID/GH via the contractor. Final Report The evaluation team, via the contractor, shall deliver five printed copies of the final report. The final report will use the following format: • An Executive Summary (no more than 5 pages) containing a clear, concise summary of the most critical elements of the report, including the commendations. • Table of Contents EXTERNAL EVALUATION OF MEASURE DHS 43 • Body of the report (no more than 30 pages), which includes: o Purpose of the evaluation o Team Composition o Methodology o Findings based on evidence o Conclusions drawn from the findings o Recommendations based on the evaluation’s findings and The Final Report will also include, but is not limited to, the following appendices: • Evaluation SOW • List of documents reviewed • List of key informants interviewed and their contact information (institution or organization) • Data collection instruments developed • Countries/sites visited • More detailed discussions of methodological or technical issues, as appropriate • A PowerPoint presentation on the results of the evaluation The body of the final report should be no more than 40 pages total, not including the annexes. If necessary, supporting data may be included in appendices, the length of which should be discussed and approved by the USAID DHS management team. The Team Leader is responsible for the content of the final report. If there is disagreement among the team members conducting the evaluation, the Team Leader should have the final decision, with dissenting opinions provided as footnotes or as an appendix. The final report should be completed no more than one week after comments from the USAID DHS management team are received. The contractor will have the report edited, formatted, and printed within approximately 15 days of receiving final USAID approval of the content. Due to procurement-sensitive information, parts of or this entire document may not be distributed outside of USAID. The contractor may prepare a separate internal USAID Memo that includes Future Directions and any recommendations that are procurement sensitive information so that the main report can be released as a public document. The contractor will make the main report available through the Development Experience Clearinghouse and its website www.dec.org. IX. Points of Contact The contacts on the USAID DHS management team include: Jacob Adetunji, COTR, GH/PRH/PEC (jadetunji@usaid.gov; 202-712-4095); Noah Bartlett, TA, GH/OHA/SPER (nbartlett@usaid.gov; 202-559-0492); YJ Choi, TA, GH/PRH/EC (ychoi@usaid.gov; 202-712-5841); Rachel Lucas, TA, GH/PRH/PEC (rlucas@usaid.gov; 202-712-4688); Lisa Maniscalco, TA, GH/HIDN/HS (lmaniscalco@usaid.gov; 202-712-1018) Madeleine Short, TA, GH/PRH/PEC (mshort@usaid.gov; 202-712-5904) EXTERNAL EVALUATION OF MEASURE DHS 44 B. List of Respondents Reached via E-survey Country/Mission Mission POC for DHS Email 1 Afghanistan Susan Brock subrock@state.gov 2 Angola Bart Bruins bbruins@usaid.gov 3 Armenia Astghik Grigoryan agrigoryan@usaid.gov 4 Benin Milton Amayun mamayun@usaid.gov 5 Burkina Faso Christian Fung cfung@usaid.gov 6 Burundi Donatien M Ntakarutimana NtakarutimanaDM@state.gov 7 Cameroon Sheila Mensah smensah@usaid.gov 8 Côte d’Ivoire Brian Howard howardb@CI.CDC.GOV 9 Dominican Republic Damani Goldstein dgoldstein@usaid.gov 10 DRC Congo Joshua Karnes jkarnes@usaid.gov 11 East Timor Teodulo Ximenes tximenes@usaid.gov 12 Egypt Shadia Attia sattia@usaid.gov 13 Ethiopia Meri Sinnitt msinnitt@usaid.gov 14 Ghana Peter Wondergem pwondergem@usaid.gov 15 Guatemala Judith Timyan jtimyan@usaid.gov 16 Guinea Daniele Nyirandutiye DaNyirandutiye@usaid.gov 17 Haiti Stephane Morisseau smorisseau@usaid.gov 18 Honduras David Castellanos dcastellanos@usaid.gov 19 India Sheena Chhabra schhabra@usaid.gov 20 Indonesia Antoinette Tomasek atomasek@usaid.gov 21 Jordan Ali Arbaji aarbaji@usaid.gov 22 Kenya Washington Omwomo womwomo@usaid.gov 23 Kyrgyzstan Sholpan Makhmudova smakhmudova@usaid.gov 24 Laos PDR Thomas D'Agnes dagnestr@state.gov 25 Lesotho Christina Lau laut@state.gov 26 Liberia Randolph Augustin raugustin@usaid.gov 27 Madagascar Voahirana Rabemanotrona vrabemanotrona@usaid.gov 28 Mali Bethanne Moskov Bmoskov@usaid.gov 29 Mozambique Matthew Rosenthal mrosenthal@usaid.gov 30 Namibia Susna De sde@usaid.gov 31 Nepal Deepak Paudel DPaudel@usaid.gov 32 Nigeria John Quinley jquinley@usaid.gov 33 Pakistan Nora Madrigal Nmadrigal@usaid.gov 34 Peru Luis Seminario lseminario@usaid.gov 35 Rwanda Nancy Godfrey ngodfrey@usaid.gov 36 Senegal Izetta Simmons isimmons@usaid.gov EXTERNAL EVALUATION OF MEASURE DHS 45 37 Tajikistan Sholpan Makhmudova smakhmudova@usaid.gov 38 Uganda Patrick Okello POkello@usaid.gov 39 Zambia Justus Kamwesigye jkamwesigye@usaid.gov EXTERNAL EVALUATION OF MEASURE DHS 46 C. PMI Contacts Reached via E-survey Country First and Last Name Email 1 Angola Jules Mihigo jmihigo@usaid.gov 2 Benin Emile Bongo ebongo@usaid.gov Abdou Salam Gueye sgueye@usaid.gov 4 DRC Jose Tchofa jtchofa@usaid.gov 5 Ethiopia Richard Reithinger RReithinger@usaid.gov 6 Joseph Malone Jmalone@usaid.gov 7 Ghana Lisa Kramer lkramer@usaid.gov 8 Paul Psychas ppsychas@usaid.gov 9 Guinea rwhanchet@usaid.gov 10 Kenya Kaendi Munguti kmunguti@usaid.gov 11 Gladys Tetteh gtetteh@usaid.gov 12 Liberia Noe Rakotondrajaona NRakotondrajaona@usaid.gov 13 Filiberto Hernandez FHernandez@usaid.gov 14 Madagascar afinlay@cdc.gov 15 Malawi Katherine Wolf kwolf@usaid.gov 16 Jessica H. Oyugi joyugi@mw.cdc.gov 17 Mali Sixte Zigirumugabe joyugi@mw.cdc.gov 18 Mozambique Juliette Morgan jmorgan1@usaid.gov 19 Nigeria Kassahun Belay kbelay@usaid.gov 20 Rwanda Roopal Patel rpatel@usaid.gov 21 Senegal Debbie Gueye dgueye@usaid.gov 22 Robert Perry rperry@usaid.gov 23 Tanzania Jessica Kafuko jkafuko@usaid.gov 24 mcelroyp@tz.cdc.gov 25 Uganda Gunawardena Dissanayake GDissanayake@usaid.gov 26 Sussann Nasr Snasr@usaid.gov 27 Zambia Oliver Lulembo olulembo@usaid.gov 28 Allen Craig acraig@usaid.gov 29 Zimbabwe Peter Halpert (HPN Officer) phalpert@usaid.gov 30 Mekong wstinson@usaid.gov EXTERNAL EVALUATION OF MEASURE DHS 47 D. List of Documents Reviewed File name Description Status of DHS Reports A word document summarizing the status (as of February 14, 2012) of current DHS Analytical Studies, Comparative Reports, Methodological Reports, Qualitative Research Studies, and Working Papers. Summary of Phase III Dissemination Activities and Products A word document summarizing dissemination activities and products to-date under Phase III. Award Fee Reports Award Fee Report Year 1 Folder includes a PDF of the Award Fee Report for Year 1 (Award Fee Report Year 1_2009), with the following key information: Criteria for Results 1.1 Country plans developed and approved for 3 countries 1.2 Draft version of DHS: The Complete Toolkit 1.3 Revised core questionnaires 1.4 Five gender-focused activities implemented Criteria for Quality 2.1 DHS data quality assurance procedures 2.2 Draft of minimum core DHS Household and Individual Woman’s Questionnaires 2.3 Training of trainer materials designed to train Macro staff who will work on surveys using PDAs for data collection Criteria for Timeliness 3.1 First year work plan submitted within 30 days after contract award 3.2 Report on Trends in the durations of fieldwork and report preparation Criteria for Management 4.1 Three meetings held with MEASURE DHS Phase III partners 4.2 Management tracking system (MEMOS) improved Criteria for Cost Control 5.1 Funding commitments of $2,500,000 received from other donors for MEASURE DHS Phase III 5.2 Two examples of cost savings Criteria for Collaboration 6.1 Participated in 8 technical consultations with international agencies and CAs 6.2 Three activities completed or in implementation stage in collaboration with other MEASURE partners and CAs Award Fee Report Year 1 Partners Meeting Folder contains all presentations from Partners Meetings 1, 2, and 3 (held on October 15, 2008, March 30, 2009), and August 24, 2009). EXTERNAL EVALUATION OF MEASURE DHS 48 Presentations Award Fee Reports Award Fee Report Year 2 Folder includes a PDF of the Award Fee Report for Year 2 (Award Fee Report Year 2_2010), with the following key information: Criteria for Results 1.1 Country plans developed and approved for 6 countries 1.2 Revised core DHS tabulation plan submitted to USAID/W 1.3 Technical review of the SPA instrument is completed 1.4 Workshop for visiting fellows conducted 1.5 Capacity building strategy for the project is developed and submitted to USAID/W for approval 1.6 The health expenditure module is field-tested 1.7 Five (5) gender- or other special audience-focused activities will be implemented, with at least 3 activities implemented in DHS participating countries Criteria for Quality 2.1 A report submitted on an assessment of the quality of birth history data collected in two countries with closely spaced surveys 2.2 SPSS tables developed to crosscheck CSPro tables for data quality assurance 2.3 Prepare and submit a questionnaire for obtaining feedback from key partners in countries that completed a DHS 2.4 Technical working consultation on scale up of the continuous survey held Criteria for Timeliness 3.1 Third-year work plan submitted on determined due date 3.2 Report documenting the impact of biomarker testing protocol review and approval processes on survey activities Criteria for Management 4.1 Two meetings held with MEASURE DHS partners 4.2 Review DHS experience with use of private sector organizations as local implementing agencies 4.3 MEASURE DHS project in-service training on GIS and IRB procedures conducted to build staff capacity to support these areas MEASURE DHS Award Fee Performance Criteria ii Report for Year 2 Criteria for Cost Control 5.1 Funding commitments of $5,000,000 received from other donors for MEASURE Phase III DHS 5.2 Comparative cost analysis submitted on the use of computer-assisted approaches in DHS surveys Criteria for Collaboration 6.1 Participated in eight (8) technical consultations with international agencies and CAs 6.2 Three (3) activities completed or in implementation stage in collaboration with other MEASURE partners and CAs 6.3 Report submitted on experience with uploading of CDC/RHS indicator data on the MEASURE DHS website EXTERNAL EVALUATION OF MEASURE DHS 49 Award Fee Report Year 2 Annex CD Files Folder contains separate PDFs for the following resources: Criterion 1.2: Guidelines for the MEASURE DHS Phase III Main Survey Report Criterion 1.3: SPA Technical Report Criterion 1.4: Agenda and Presentations from Visiting Fellows Workshop Criterion 1.6: Report on the Pretest of the MEASURE DHS Health Expenditures Module Criterion 4.1: Agenda, Attendee List, and Presentations from DHS Partner Meetings Criterion 4.3: GIS and IRB Training Presentations Award Fee Reports Award Fee Report Year 3 Folder includes a PDF of the Award Fee Report for Year 3 (Award Fee Report Year 3_2011), with the following key information: Criteria for Results 1.1 Revised core SPA questionnaire submitted to USAID/W 1.2 Finalized DHS curriculum posted on the website and elements of the curricula used or planned to be used by at least one faculty member from three different universities in developing countries 1.3 First phase of a DHS junior faculty fellowship program launched in E. Africa 1.4 The Senegal continuous survey launched 1.5 A report on the calculation of concurrent partnerships from DHS data completed and submitted to USAID/W 1.6 The Mali qualitative family planning study completed and a report detailing research questions, methods, findings, and dissemination plans submitted Criteria for Quality 2.1 Report prepared reviewing changes in DHS sample sizes and the drivers and data quality implications of increasing sample sizes 2.2 Report looking at procedures involved and compared the results of rapid diagnosis tests (RDTs) and microscopy results in malaria indicator surveys (MIS) under DHS Phase III prepared 2.3 Report reviewing measures to ensure data quality in surveys using computer-assisted approaches prepared Criteria for Timeliness 3.1 Project technical documents submitted in a timely manner 3.2 Report prepared documenting adherence to survey timetable among four selected surveys which started fieldwork under this contract and held the national seminar by September 1, 2011 Criteria for Management 4.1 Country plans for all countries that have completed fieldwork by August 31 under this contract uploaded to MEMOS and made accessible to USAID/W 4.2 Report prepared on advantages and disadvantages of using disbursing agents to aid in the local management of implementing of surveys in Phase III 4.3 Report prepared on project management processes put in place to enhance quality across surveys MEASURE DHS Award Fee Performance Criteria ii Report for Year 3 Criteria for Cost Control 5.1 Funding commitments of $7,500,000 received from other donors for EXTERNAL EVALUATION OF MEASURE DHS 50 MEASURE Phase III DHS 5.2 Four examples of cost savings provided Criteria for Collaboration 6.1 Participated in six technical consultations with CAs and international agencies 6.2 Three (3) activities completed or in implementation stage in collaboration with other MEASURE partners and CAs Country Plans Folder includes the following resources: All 2011 Country Plans Combined Cambodia Country Plan 2011 Malawi Country Plan 2011 Rwanda Country Plan 2011 Tanzania Country Plan 2011 Work Plans Folder includes work plans for years 1-3. The work plan for year 4 will be provided shortly. First Year Work Plan Second Year Work Plan Third Year Work Plan Phase III Trip Reports Folder includes 60 separate PDFs, one per country, with all trip reports for that country during Phase III. (Note that the Malawi PDF has been updated). DHS Curriculum Folder includes the following resources: Instructor’s Guide PDF Module 1 PDF Module 2 PDF Module 3 PDF Module 4 PDF Module 5 PDF Module 6 PDF Module 7 PDF Module 1 PPT, zipped Module 2 PPT, zipped Module 3 PPT, zipped Module 4 PPT, zipped Module 5 PPT, zipped Module 6 PPT, zipped Module 7 PPT, zipped Elements of the DHS Toolkit Elements of the DHS Toolkit are included in 4 subfolders: Data Collection, Biomarker, Data Processing, and Data Dissemination. For a full listing of the documents included in these folders, please refer to following the Excel sheet EXTERNAL EVALUATION OF MEASURE DHS 51 located in the primary Toolkit folder: “List of Included Toolkit Documents.” Results Reports Folder includes the following resources: Phase III, 2008-9 Results Reporting Phase III, 2009-10 Results Reporting Phase III, 2010-11 Results Reporting Results Reports Ultra Fabs Folder includes the following resources: Ultra Fabs 2008 Ultra Fabs 2009 Ultra Fabs 2010 Ultra Fabs 2011 Tanzania Folder includes the following resources: List of Tanzania Materials Tanzania Contact List Phase III Tanzania Trip Reports Tanzania Dissemination Activity Summary Tanzania Dissemination Products Folder includes the following resources: Factsheet Final Child Health Booklet Final Youth Booklet Gender Based Violence Factsheet Key Findings Malaria Mini Poster Micronutrient Flyer Reading DHS Tables (for 2010 TDHS) TZ_PMI_abstract Tanzania National Seminar PowerPoints Folder includes the following resources: Characteristics of Respondents and Households Child Health Family Planning Fertility determinants HIV knowledge, attitudes, and behavior Malaria Maternal Health Methodology Mortality EXTERNAL EVALUATION OF MEASURE DHS 52 Nutrition Women’s Status (GBV, and FGC) Malawi Folder includes the following resources: Malawi Contact List Malawi Phase III Trip Reports Malawi Dissemination Products Folder includes the following resources: 2010 MDHS Reading and Understanding DHS Tables Malawi 2010 DHS General Factsheet Malawi 2010 DHS HIV Fact Sheet Malawi 2010 DHS Key Findings Malawi 2010 DHS Wall Chart Malawi National Seminar PowerPoints Folder includes the following resources: Characteristics of Respondents and Households Child Health Family Planning Fertility determinants HIV knowledge, attitudes, and behavior HIV Prevalence Malaria Maternal Health Methodology Mortality Nutrition Women’s Status and GBV Note: PowerPoint presentations tailored for each region were also prepared by DHS staff for use during Regional Dissemination Seminars held in the Central, Northern, and Southern regions. The Seminars themselves were organized and implemented without DHS participation. Due to space constraints, these 3 series of PowerPoints are not included, but they are available upon request. Cambodia Folder includes the following resources: Cambodia Contact List Cambodia Phase III Trip Reports Cambodia Dissemination Products Folder includes the following resources: Cambodia DHS 2010 Fact Sheet Cambodia DHS 2010 Key Findings EXTERNAL EVALUATION OF MEASURE DHS 53 Cambodia DHS 2010 Reading DHS Tables Cambodia National Seminar PowerPoints Folder includes the following resources: Intro and Methodology Respondents and Households Family Planning Fertility Child Health Maternal Health Women’s Status HIV Knowledge, Attitudes, and Behavior Accident and Injury Mortality Nutrition 1 Nutrition 2 Bangladesh Folder includes the following resources: Bangladesh Contact List Bangladesh Media Activities Summary Final Report-BDHS Dissemination and Capacity Building Status Report 2011 Bangladesh DHS Jan 2012 Bangladesh Phase III Trip Reports Bangladesh Extended Dissemination Materials for 2007 BDHS Folder includes the following resources: Bangladesh Nutrition Booklet Bangladesh Handout Core Presentation DHS PoP Nutrition Brochure How Healthy are Bangladesh Families? 4-pager Sylhet Fact Sheet Bangladesh Bangladesh Journalist Workshop Folder includes the following resources: Bangladesh March 2010 Journalist WS Trip Report Participant Contact List Participants’ Evaluation Pre-Post Knowledge Assessment Results Summary Presentation on Journalist Workshop (PPT) Survey-Related Capacity Folder includes the following two excel sheets: EXTERNAL EVALUATION OF MEASURE DHS 54 Building Phase III Measure DHS Survey-Related Capacity Building Training Summary Worksheet (updated with a final summary sheet) Phase III Measure DHS Special Trainings Summary Worksheet (updated with details on the Bangladesh 2010 Journalist Workshop) Capacity Building Core Curriculum Workshop Documentation Separate subfolders in this folder contain available documentation for the November 2010 Kenya Workshop, December 2011 Malawi Workshop, October 2011 Malawi Workshop, July 2011Tanzania Workshop, and August 2011 Uganda Workshop. Documentation includes trip reports, participant lists, and/or evaluations wherever possible. (Updated with evaluation materials for the November 2010 Kenya Workshop). Capacity Building Fellows Workshop Documentation Separate subfolders contain available documentation for the 2010 Short Term Fellows Workshop (Round 2) and the 2011 Junior Faculty Fellows Workshops (Round 3). Documentation includes participant lists, agendas, and/or evaluations wherever possible. Capacity Building Specialized Training Workshop Documentation Separate subfolders contain available documentation for the Zimbabwe 2009 Workshop, Bangladesh 2010 Journalist Workshop, Indonesia 2010 Workshops, Albania 2011 Workshop, Kenya 2011 Workshop, Uganda 2011 Workshop, and Regional 2011 Journalist Workshop. Documentation includes trip reports, participant lists, agendas, and/or evaluations wherever possible. EXTERNAL EVALUATION OF MEASURE DHS 55 E. List of Key Informants Interviewed, by Organization ICF International Staff, Calverton, Headquarters: (14) Adrienne Cox, ICF Albert Themme, ICF Alfredo Aliaga, ICF Ann Way, ICF Annie Cross, ICF Bernard Barrere, ICF Binyam Woldemicheal, ICF Guillermo Rojas, ICF Martin Vaessen, ICF Monica Kothari, ICF Sri Poedjastoeti, ICF Sunita Kishor, ICF Paul Ametepi, ICF Fred Arnold, ICF DHS Management Team: (6) Jacob Adetunji, USAID/GH/PRH/PEC Lisa Maniscalco, USAID/GH/HIDN/HS Madeleine Short, USAID/GH/PRH/PEC Noah Bartlett, USAID/GH/OHA/SPER Rachel Lucas, USAID/GH/PRH/PEC Y.J. Choi, USAID/GH/PRH/PEC Other USG (24): Bob Emrey, USAID/GH/HIDN/HS Elizabeth Schoenecker, USAID/GH/PRH/PEC Alex Rowe, CDC Ishrat Husain, USAID/AFR/SD/HT John Borrazzo, USAID/GH/HIDN/MCH EXTERNAL EVALUATION OF MEASURE DHS 56 Kathryn Panther, USAID/GH/PRH/SDI Kelly Saldana, USAID/GH/HIDN Krista Stewart, USAID/GH/OHA/SPER Misun Choi, USAID/GH/HIDN/ID Neal Brandes, USAID/GH/HIDN/NUT Paul Bouey, OGAC Paul Holmes, USAID/EE/DGST Gary Cook, USAID/ME/TS Paul Stupp, RHS Surveys, CDC/Atlanta S. Rene Salgado, USAID/GH/HIDN/ID Roxanna Rogers, USAID/GH/OHA Paul Mahanna, USAID/GH/OHA Sally Abbott, USAID/GH/HIDN/NUT Scott Radloff, USAID/GH/PRH Ellen Starbird, USAID/GH/PRH Sarah Harbison, USAID/GH/PRH Steve Yoon, CDC Susan Thollaugh, USAID/LAC Karen Cavanaugh, USAID/GH/HIDN/HS Other/U.S. based (6): Amy Tsui, JHU/SPH Holly Newby, UNICEF Paul Delay, UNAIDS/Geneva Richard Cibulskis, WHO/Geneva Robert Stienglass, JSI Vinod Mishra, UN Population Division Bangladesh (25) Jan Borg, AUSAid Shahrouk Safi, AUSAid Kanta Jamil, USAID Mission EXTERNAL EVALUATION OF MEASURE DHS 57 Khadijat Mojidi, USAID Mission Dr Farhana Ahmad, White Ribbon Alliance Dr Ahmed al-Kabir, RMT Shisir Morel, Bangadesh Daily Newpaper Hasib, BD24.com journalist Faud Pasha, Mitra and Associates Shelina Afroza, Mitra and Associates Anisul Awwal, Mitra and Associates Mohammed Alam, Mitra and Associates Shahin Sultana, Mitra and Associates SN Mitra, Mitra and Associates Shahidul Islam, Mitra and Assocaites Karar Zunaid Ahsan, World Bank Farhana Afroz, Management Resources Development Initiative (MRDI) Toslim Uddin Khan, Social Marketing Company (SMC) Nitai Chakraborty, Department of Statistics, Biostatistics, and Informatics, University of Dhaka Dr. Shamim Hayder Talukder, Eminence Center for Health and Development Intelligence Dr. Nurun Nabi, Department of Population Sciences, University of Dhaka Arthur Erken, UNFPA Dr. Kim Streatfield, Center for Population, Urbanization, and Climate Change, International Center for Diarrheal Disease Research of Bangladesh Dr. Laura Reichenbach, Center for Reproductive Health, International Center for Diarrheal Disease Research of Bangladesh Dr. Shams El Arifeen, Centre for Child and Adolsecent Health, International Center for Diarrheal Disease Research of Bangladesh Cambodia (22) Savina Ammassari, UNAIDS Tara Milani, USAID Mission Prateek Gupta, USAID Mission Chanta Chak, USAID Mission EXTERNAL EVALUATION OF MEASURE DHS 58 Yi Soktha, UNFPA Phan Chanda, NIS H.E. San Sy Than, NIS Saint Lundy, NIS Dr Sao Sovanratnak, WHO Mao Bonsoth, University of Health Sciences Kim Net, NIS Joel Conkle, UNICEF Usha Mishra, UNICEF Tung Rathavy, Ministry of Health Mondol Loun, Health Information and Planning, MOH Kamphor Meng, NIS Sarthi Acharya, UNICEF Uy Bossadine, NIS H.E. Hor Darith, World Bank Emre Özaltin, World Food Program Kurt Burja, World Food Program Sheryl Keller, Independent Consultant Malawi (26) Dr. Benson Chilima, Community Health Sciences Unit (CHSU), MOH Mandigore Yassin, Community Health Sciences Unit (CHSU), MOH Kundai Moyo, Community Health Sciences Unit (CHSU), MOH Nellie Kabonde, Community Health Sciences Unit (CHSU), MOH Andreas Jahn, Community Health Sciences Unit (CHSU), MOH Dorothy Nyasulu, UNFPA Thomas Chataghalala Munthali, UNFPA Felicitas Zawaira, WHO Harriett Chanza, WHO Glory Mkandawire, JHU/BRIDGE Project William Dothi, JHU/BRIDGE Project Peter Roberts, JHU/BRIDGE Project EXTERNAL EVALUATION OF MEASURE DHS 59 Benjamin Kaneka, Department of Population Studies, Chancellor College, University of Malawi James Kaphuka, Department of Population Studies, Chancellor College, University of Malawi Charles Machinjili, National Statistical Office Mercy Kanyuka, National Statistical Office Angela Msosa, National Statistical Office, Demography and Social Statistics Division Mylen Mahowe, National Statistical Office, Demography and Social Statistics Division Tiope Mleme, National Statistical Office, Demography and Social Statistics Division Medson Makwemba, National Statistical Office, Demography and Social Statistics Division Richard Phiri, National Statistical Office, Demography and Social Statistics Division Isaac Dambula, Ministry of Development Planning and Cooperation Blackson Matatiyo, National AIDS Commission, Malawi Chris Moyo, Monitoring and Evaluation, Ministry of Health Koorosh Raffii, Monitoring and Evaluation, UNICEF/Malawi Nellie Wadonda-Kubondo, CDC/Malawi Anteneh Worku, USAID/Malawi Tanzania (18) Erasmo Malekela, USAID/Tanzania Dr. Raz Stevenson, USAID/Tanzania Dr. Julitta Onabanjo, United Nations Population Fund (UNFPA) Dr. Rutasha Dadi, United Nations Population Fund (UNFPA) Dr. Rita Noronha, United Nations Population Fund (UNFPA) Samweli Noronha, United Nations Population Fund (UNFPA) Victoria Chua, UNICEF Mrs. Aldegunda Komba, Department of Social and Demographic Statistics, National Bureau of Statistics Mr. Mlemba Abassy, Social and Demographic Statistics Department, National Bureau of Statistics Mr. Emillion Karugendo, Social and Demographic Statistics Department, National Bureau of Statistics Mr. Deogratius (Deo) Malamsha, Social and Demographic Statistics Department, National Bureau of Statistics Mrs. Renata Mandike, Tanzania Malaria Control Board Dr. Peter McElroy, Centers for Disease Control and Prevention EXTERNAL EVALUATION OF MEASURE DHS 60 Dr. Aroldia Mulokozi, Tanzania Commission for AIDS Professor Mecky Matee, Department of Microbiology and Immunology, School of Medicine, Muhimbili University, Department of Health and Allied Sciences (MUHAS) Vincent Assey, Biomarker Analyst, Tanzania Food Science and Nutrition Department Dr. Neema Rusibamayila, Reproductive and Child Health Division, Ministry of Health and Social Welfare Dr. Rik Peeperkorn, Netherlands Embassy EXTERNAL EVALUATION OF MEASURE DHS 61 F. Data Collection Instruments Developed F.1. Question line for Macro management and senior leadership Task 1.1: Does the current DHS survey suite meet the needs? 1. In an era of increasing demands for population based data, what should be the products of the Measure DHS project? • Are you delivering more than is used or needed? • Are you offering more than you can deliver efficiently and effectively? • Should you be delivering something more or something different? 2. What strategies have you put in place to keep an element of control over the increasing demands for data from DHS population surveys at the US government or global level? • Can you give me an example of what has worked? • Can you give me an example where efforts have been unsuccessful? 3. Is it Macro’s responsibility to coordinate with other providers of data either globally or in-country to avoid duplication? Is there a need for another mechanism to work through these demands within country or are the current arrangements (such as the technical committee) sufficient? 4. If we put this in a broad context, beyond negotiating the content of a survey with the DHS suite, in what ways if any is Macro involved in coordinating with other providers of data either globally or in-country to avoid duplication? 5. Who are the heaviest users of DHS data from your perspective? What results do they use most often? 6. Who are the most “important” users of DHS data from your perspective? 7. We have seen the documentation from the Award documents, but can you explain the trends in the degree of support, cash or in-kind, coming from the government or local donors? Have there been any recent trends in the level of the support and what are the drivers of that trend? 8. Overall, what are the DHS results that are in greatest demand by your different stakeholders (e.g. fertility, vaccination coverage, HIV seroprevalence, other)? 9. Which outputs / results cause the most controversy? What is the controversy about? 10. What outputs are the most in demand? (e.g. final reports, summaries, data sets, your web profiles etc.) In this funding cycle have you changed some of the output formats to meet user demands? Do you undertake some marketing research exercise to determine what products are needed? Tell us about it? Task 1.2: How successfully is the contractor providing technical assistance in overall survey content, design and implementation? EXTERNAL EVALUATION OF MEASURE DHS 62 We want to understand the factors that determine the level of TA given in a country and to explore the efficiency and effectiveness of that TA. 11. Overall, what is the process used to determine the types and volume of TA to be provided in a specific country. • How do you ensure that the range and intensity of TA to local implementing agencies is appropriately targeted? • In what ways are host country partners involved in that decision process? 12. Is a TA Plan developed for each survey/country? If so how is it articulated? Reviewed/approved? Monitored/evaluated? Get written examples. 13. How are decisions made regarding trip duration and number of countries to be visited? 14. What is the travel approval process at DHS? 15. DHS documentation makes reference to “different models” of technical assistance. Please describe these different models, and how they are applied 16. What are the lessons learned from the application of different models of providing technical support and survey implementation in different regions, such as country residential advisors, regional advisors, HQ￾based TA, south-to-south exchange of TA, etc. How is the decision made to employ a specific model? 17. To what extent has the private or non-government sector been used for implementing aspects of the DHS suite of surveys? 18. What have been the main successes and challenges of working with private / nongovernment sector implementing partners? • In particular, do they have different TA needs? What lessons learned, if any, may be applied to the public sector? 19. What specific ways, if any, might TA be provided more efficiently, without sacrificing effectiveness? 20. We would like to have data showing the amount of TA in person days for each survey completed during this funding period. If data is accessible for a longer period we would like to have historical trends. 21. Are there areas of cost savings that can be identified and implemented, especially in relation to: • Management of survey implementation process • Training of interviewers • Transportation and survey logistics 22. How would you rate the overall efficiency of TA provided by DHS, and why? 23. What specific ways might TA be provided more efficiently, without sacrificing effectiveness? 24. Which stakeholders, if any, are concerned about the level of TA? What are their concerns? In what ways, if any, does this resistance pose a risk to the DHS program? EXTERNAL EVALUATION OF MEASURE DHS 63 25. Now we would like to get into specifics. In your experience of the most recent rounds of the Measure DHS surveys, which are the areas in which you believe that TA is most important and should not be sacrificed and what are the areas in which TA is actually provided? TA is … 1 = usually not provided 2 = minor amount of TA provided 3 = large investment in TA is provided What is the usual method of providing TA? (FIFO, Resident Advisory, via phone or email?? Was that level of TA appropriate for local needs? 1 = insufficient 2= sufficient, well targeted 3 = more than was required to ensure a quality survey (ask for more information) The decision making process of determining which DHS surveys to meet government and donor needs in country Questionnaire adaptation Sampling Interviewer training Management of field operations (including bio markers) Management of field operations (Excel bio markers) Data quality assurance (questionnaire validation) Data processing Generation of preliminary country reports Generation of final country reports Dissemination Advocacy and research translation Secondary analysis of DHS data Task 2.1: Are current capacity building strategies and activities meeting local needs? 26. Based on your experience and your perception of the principal reasons for the DHS suite of surveys, do you think should be the goal for capacity building related to Measure DHS should be to (score 1 to 5 with 5 the highest priority). EXTERNAL EVALUATION OF MEASURE DHS 64 Build local capacity to run DHS surveys with minimal TA? Build local capacity to use DHS results for program planning? Build local capacity to run a wide range of quality population-based services in response to country need? Build local capacity to undertake DHS related tasks to an appropriate standard with the guidance and support of the contracting agency? 27. How would you assess that ability of the current DHS program to meet the capacity building priorities you have mentioned? 28. What have been the successes? Give us an example or two of how the program has resulted in improved in-country capacity. 29. Do you set capacity building objectives for a country at the beginning of the survey planning process? Is this focused on ensuring the quality of the survey or are there other objectives? 30. What are the incentives for country managers to build capacity within countries? 31. In looking at select countries that have had multiple rounds of DHS (and other Macro-sponsored surveys), how has the role of the local implementing partner(s) changed or not changed over time (i.e. what capacity have they demonstrated over time)? 32. We want to be very clear on what is being done in capacity building and where you believe that the priorities should lay: Current degree of effort to build capacity 1 = Little or none 2= Some effort 3 = Substantial effort (note methods used) Building capacity should be a priority 1 = not important 2 = moderately important 3 = Very important The decision making process of determining which DHS surveys to meet government and donor needs in country Questionnaire adaptation Sampling Interviewer training Management of field operations (including bio markers) Management of field operations (Excel bio markers) EXTERNAL EVALUATION OF MEASURE DHS 65 Data quality assurance (questionnaire validation) Data processing Generation of preliminary country reports Generation of final country reports Dissemination of study findings Advocacy and research translation using survey research Secondary analysis of DHS data and other population survey data Ability to independently design and conduct population based surveys with bio markers Ability to independently design and conduct reproductive and MCH population based surveys Ability to independently design and conduct RCT or quasi experimental population based research designs Task 2.2: What capacity building approaches might increase the contractor’s performance in the area of local institutions and individuals in survey design / management and data analysis and use? 33. What is the professional development / capacity building processes internal within Marco? If resistant say that best practice in capacity building was that it was a whole of organization approach. 34. How does DHS keep informed about emerging state-of-the-art approaches for capacity building that might be applicable to DHS? (Prompts: staff member assigned task; attend conferences/workshops on the subject; review web/literature; investigate what other similar organizations are doing, outsourcing a review and plan etc.) 35. What are lessons learned from the application of different models of providing technical support and survey implementation in different regions, such as country residential advisors, regional advisors, HQ￾based TA, south-to-south exchange of TA, etc. Is there anything else I should know for this review that you have not discussed already? Is there anyone you suggest we interview as part of this review who might not have already been identified? F.2 Question line for Missions and Other Stakeholders in Country Task 1.1 Does the current DHS suite meet the needs? EXTERNAL EVALUATION OF MEASURE DHS 66 36. What is the level of interest in the DHS among the other donors and the government? Has there been support, cash or in-kind, coming from the government or local donors? 37. Were you involved in the discussions about what would be in the surveys that were done in the country? If yes, • Were you satisfied with the process of making the decisions? • Where you satisfied with the final decision? • What would you have liked to have happened? 38. Which other groups were involved in the planning process? 39. What was the role of the DHS representative (country manager) in these discussions? 40. In what ways if any is Macro involved in coordinating with other providers of data either globally or in￾country to avoid duplication? 41. In an era of increasing demands for population based data, what should be the products of the Measure DHS project? • Are they offering more than they should be delivering? • Should they be delivering something more or something different? 42. In what ways, if any, do the current MEASURE DHS outputs meet needs for monitoring and evaluation your field / country? 43. Who are the major users of DHS data in this country? How are they used (please give examples) • Basic demographic indicators (age structure, fertility) • Indicators of progress on impact and outcomes • Program planning • Quasi experimental evaluations 44. How do you use the DHS results? What results do you use most often? Please tell me about the last time you used DHS data? 45. In recent surveys (last 2-3 years) have any outputs / results cause controversy? What is the controversy about? 46. What outputs of the DHS do you use: summary papers, briefing notes? Would you like the information in another form? Can you think of another repository of data that has features that you like that would be appropriate for Measure DHS to adopt? 47. What is a question that you have had recently wished you had the data to answer? (Note the point is whether people want things that DHS doesn’t have or if they just don’t realize what is in the DHS suite.) 48. Do you have any concerns about the quality or the relevance of some aspects of the DHS survey suite and their results? If yes, are these concerns about data quality, relevance of indicators, topics or variables not covered – i.e. too superficial, too much data that is never used? EXTERNAL EVALUATION OF MEASURE DHS 67 Task 1.2: How successfully is the contractor providing technical assistance in overall survey content, design and implementation? We want to understand the factors that determine the level of TA given in a country and to explore the efficiency and effectiveness of that TA. 49. What is your assessment of the technical capacity of the local implementing agency? How capable of designing, implementing, analyzing and reporting on a population based survey such as the DHS suite of surveys? 50. Does the local implementing agency undertake other surveys for the host government, USAID, other donors or NGOs? • What is the reputation of that work? • Have well-founded concerns about quality been expressed? 51. Overall, thinking about the last survey, was the range and intensity of TA provided by Macro for the most recent survey appropriately targeted? That is, was it too little (more was needed), too much (unnecessary trips), or just right? 52. What was the decision making process about how much TA was provided during the recent surveys? • Was a TA plan prepared and shared with your organization? • In what ways, if any did your organization participate in those decisions? 53. Knowing the local situation, could the TA provided have been delivered as effectively or more effectively in a different way? • Remotely through various telecommunications methods? • Through a local technical expert? • In any other way? 54. Had another method been proposed? What were the reasons given for that not being done? 55. Has there been any involvement of the private sector, universities or NGOs implementing aspects of a DHS survey in this country? What are the advantages and disadvantages of their involvement? Did they have different TA needs or provision? 56. Are there other models of TA that in your experience have worked well in this or a similar setting and could be applied to the DHS? What are the features of these other models? What is better about these models: more efficient? More effective? More appropriate? 57. Are there TA models which are not appropriate in this setting which we should know about to inform this review? Task 2.1: Are current capacity building strategies and activities meeting local needs? EXTERNAL EVALUATION OF MEASURE DHS 68 58. In what ways, if any, has the contractor build in-country capacity? Provide examples. • What strategies or activities were effective? • What else contributed to increased capacity? 59. In looking at select countries that have had multiple rounds of DHS (and other Macro-sponsored surveys), how has the role of the local implementing partner(s) changed or not changed over time (i.e. what capacity have they demonstrated over time)? 60. How would you rate the effectiveness of the capacity building that resulted from the last survey? What was gained (be specific about design, implementation, data management, data analysis, report writing, dissemination, data use for advocacy, monitoring, evidence-based decision making)? 61. Are USAID’s and the project’s goals and objectives vis-à-vis capacity building along the data collection continuum appropriately targeted? Is an appropriate mix of capacity building activities at the institutional/organizational and individual levels present? Task 2.2: What capacity building approaches might increase the contractor’s performance in the area of local institutions and individuals in survey design / management and data analysis and use? 62. Are there models and lessons learned from other data collection projects that this project can learn from? 63. Are there more appropriate organizations or agencies to be involved in building some aspects in capacity? Is this something that US government is funding through other mechanisms? 64. What comparable programs are happening in this country (through donors, government, universities, NGOs) to increase capacity in population based research design, implementation, analysis and use which would complement or replace the capacity building work by DHS? 65. If yes, to your knowledge has there been discussions or collaboration between that group and DHS? Is there anything else I should know for this review that you have not discussed already? Is there anyone you suggest we interview as part of this review who might not have already been identified? F.3 E-Survey Questionnaire Demographic and Health Survey Evaluation March-April, 2012 E-Survey Questionnaire MEASURE Phase III Demographic and Health Surveys MEASURE/Demographic and Health Surveys (DHS) is USAID’s flagship global health data collection project. Since its inception in 1984, the DHS has conducted more than 240 surveys in about 85 countries in Africa, Latin America, Asia, and East Central Europe. The overall objective of MEASURE DHS is improved EXTERNAL EVALUATION OF MEASURE DHS 69 collection, analysis and presentation of data to promote better use in planning, policymaking, managing, monitoring and evaluating population, health and nutrition programs. Note: In this questionnaire, the term “DHS” may refer to any of the various types of population-based surveys implemented by Macro under the MEASURE Phase III Demographic and Health Surveys. Many countries have conducted more than one type of DHS survey since 2008. Therefore, in Question 1, you are asked to specify what surveys have been completed in your country in the last four years (2008-2011). When we ask about aspects of the DHS survey we are referring to all surveys in this period. 1. What types of DHS surveys were completed in your country between 2009 and 2011? (Please check all that apply) Standard Demographic and Health Survey______ AIDS Indicator Survey (AIS) _____ Service Provision Assessment (SPA) Survey _____ Malaria Indicator Survey (MIS) _____ Interim DHS_____ Other (please specify) [Text box] 2. Overall, did these surveys meet your country’s needs for information? (Please check one) Yes, extremely well _____ Yes, for some needs _____ No, it is not very useful _____ 3. Please describe any shortcomings that reduced the usefulness of the surveys? [Text box] 4. How have you or your office used the results of the recent DHS surveys? (Please check all that apply) Reporting requirements _____ Programming programs/projects _____ Policy dialogue _____ Monitoring and evaluation of programs/projects _____ Other – please specify_____________________________________ 5. What outputs from the recent DHS surveys do you or your country office use regularly? (Please check all that apply) Final country report _____ Summary or Key Findings report _____ Dissemination posters _____ Further analysis reports _______ Complete survey data file _____ STAT Compiler on the DHS website ___ Other – please specify_________________________________ EXTERNAL EVALUATION OF MEASURE DHS 70 6. What results from the recent DHS surveys have been most essential for your program efforts? (e.g., fertility rate, contraceptive use, infant and child health, EPI coverage, HIV seroprevalence, etc.)? (Text box) 7. Who are the major stakeholders in the country who were involved in developing the recent DHS surveys in your country? (Please score as 0=No, not involved,1=Yes, but limited involvement, and 2=Yes, heavily involved) Ministry / Department of Health _____ Other host government ministries (specify) _____ USAID Mission______ Other USG Agency partners (PMI, CDC, etc.)______ National or regional universities and research institutes _____ International NGOs ______ Local NGOs _____ UNICEF _____ UNFPA _____ World Bank _____ Other stakeholders – please specify _______________________________ _____ __________________________________________________________________ 8. Overall, was Macro responsive to the information needs of these stakeholders? Yes, very responsive _____ Yes, responsive within technical or budgetary constraints _____ No, not responsive _____ Don’t know _____ 9. To the best of your knowledge, who are the major users of the most recent DHS surveys in your country? Major User Occasional User Not a User Ministry / Department of Health _____ _____ _____ Local Universities _____ _____ _____ International NGOs _____ _____ _____ Local NGOs _____ _____ _____ UNICEF _____ _____ _____ UNFPA _____ _____ _____ World Bank _____ _____ _____ USAID ______ ______ ______ Others – please specify _____ _____ _____ _______________________________ _____ _____ _____ _______________________________ _____ _____ _____ EXTERNAL EVALUATION OF MEASURE DHS 71 10. How would you describe Macro’s level of technical assistance effort during the recent DHS surveys? Extensive TA during all phases of survey operations _____ Moderate TA provided during selected phases of survey operations _____ Limited TA provided ______ No TA provided by ICF/Macro _____ Don’t know _______ 11. What have been the types of TA provided during the implementation of your last DHS? (Please rank as 1= Low TA source, 2=Moderate TA source, 3=Major TA source. Leave blank if no support provided through category) Macro staff visiting from Calverton ______ Macro resident technical advisers _____ National professionals ______ South-to-south technical consultants ______ USAID Mission staff _____ Macro emails or other virtual TA______ Other _____ 12. If your country undertakes another DHS, would you prefer that TA be provided in a different manner? Yes _____ If yes, please specify ___________________________________________________ _____________________________________________________________________________ No______ 13. In what areas has there been effort to build population and health survey capacity in your country? (Check all that apply) Improving the skills and self sufficiency of local implementing agencies and individual staff members to plan and conduct surveys and collect, process, and disseminate data _____ Building capacity of labs and lab technicians to collect, process, and analyze biomarker data______ Increasing skills among multiple audiences to interpret and use DHS results for evidence-based planning and monitoring _____ Enhancing skills among researchers and academics to analyze DHS data sets _____ No capacity building focus _____ Don’t know, no opinion _____ Other (please specify) ______ 14. Have you seen growth in capacity in this country to collect, analyze, or use data population and health survey data in the past three years? Yes, large growth Yes, some growth Yes, limited growth No growth Deterioration in capacity Don’t know EXTERNAL EVALUATION OF MEASURE DHS 72 Please give specific examples what capacity has been increased? Text Box 15. Overall, how satisfied were you with Macro’s capacity building activities in the last three years? Very satisfied, considerable improvement in capacity was observed ______ Satisfied, Macro made contributions to capacity ______ Not satisfied, Macro could have done more to build capacity _____ Very unsatisfied, Macro did little or nothing to build capacity _____ Do not know, no opinion _____ 16. Were funds allocated as specific budget items for capacity building? Yes _____ No _____ 17. If you could strengthen capacity building activities for the DHS in your country, where would you recommend efforts be focused? (Text Box) EXTERNAL EVALUATION OF MEASURE DHS 73 G. Country Selection Criteria • All countries were to have had repeat surveys • Countries were selected to reflect different capacity levels and resulting likely variance in the intensity/need for TA • Countries were selected based on use of different models of TA, using regional staff or south-to￾south, etc. EXTERNAL EVALUATION OF MEASURE DHS 74 H. Country Case Descriptions Bangladesh, Cambodia, Malawi, and Tanzania Malawi Case Study Background Malawi performed one MEASURE DHS survey, the 2010 Malawi Demographic and Health Survey (DHS), during the Evaluation period. The DHS included several biomarkers, namely anemia testing for children under five and women and HIV testing for women and men. The national sample included 27 district estimates from 27,345 households, 22,200 women, and 7,000 men. Implementation partners for the Tanzania DHS included the National Statistical Office (NSO), the Community Health Sciences Unit of the Ministry of Health (CHSU), the National AIDS Commission (NAC), and the MOH. Funding support came from the Government of Malawi, NAC, USAID, UNICEF, UNFPA, CDC, and DFID. An MIS survey in Malawi currently is underway. Evaluator Description 1. Meeting Stakeholder Needs (Task 1) The MDHS informs policy and program planning and provides data on health outcomes and disease burdens. MDHS data are widely used since they are highly relevant for measuring development outcomes in Malawi, monitoring the National Development Plan, and formulating program management and policy in the MOH and other ministries. The DHS is also a major source of data for monitoring service delivery programs for the MOH. It was used extensively in preparing the Health Sector Strategic Plan for period 2011-2016. The MDHS is an important source of data for HIV/AIDS program coverage, knowledge, the presence of risk behaviors, and estimates of the HIV prevalence rate. The NAC does its own dissemination of MDHS HIV/AIDS data at the national and sub-national levels. In 2004, 10 of 28 districts were oversampled to provide district-level estimates of HIV seroprevalence. NAC was upset that district-level estimates of HIV prevalence were not tabulated for the 2010 survey. There will be a follow-up workshop to re-visit HIV tabulations and produce an addendum to the Final Report. In terms of data utilization, respondents noted the following: (1) additional thematic policy briefs were needed to translate data into more understandable policy language, (2) more funding is needed for further analysis of data, and (3) TA is needed for capacity building in grant writing. There was also significant pressure to get the data out as quickly as possible. Numerous respondents noted that the DHS survey process took “too long.” WHO had wanted to use the 2010 data for collaborating in the development of a new health sector strategy for Malawi, but the Final Report came out too late. In Malawi, survey planning begins early. The Donor Group on Statistics is the coordinating body through which the Government announces plans for conducting the DHS. Planning begins with the details of the survey including the development of a survey protocol document and work plan. At this stage, the Ministry of Health also reviews the questionnaire from the last DHS to determine the relevance of each question and identify new, emerging data needs. For each DHS in Malawi, a National Steering Committee was formed with representation from the National Statistical Office (NSO), the Ministry of Health and Social Welfare, the Ministry of Planning, the Ministry of Community Development, Gender, and Children, NGOs, international organizations, and university leaders. EXTERNAL EVALUATION OF MEASURE DHS 75 The National Steering Committee (NSC) provides guidance on survey content, policy directives for managing survey operations, and the management of financial resources. The NSC includes representation from the NSO, MOH, Ministry of Planning, NGOs, university leaders, USAID, and DFID. The NSC is most active during the survey design and data analysis stages. When suggestions for additional questions are made, the Committee decides which ones should be included. There was agreement that a “good” level of stakeholder involvement was reached during the questionnaire development process. There were also multiple stakeholder meetings regarding survey content, but many were not clear on how final decisions regarding survey content were made. Several respondents noted that in the future, they would like to be more involved in processes for questionnaire design, training of field workers, field supervision, data editing, and further analysis. 2. Effectiveness of Technical Assistance (Task 1) The TA provided by ICF was generally considered to be satisfactory, but several respondents would have liked to see more local involvement in all stages of the survey. It was observed that the NSO could have used more TA at the HIV/AIDS testing stage, but otherwise the volume of assistance was appropriate. In Malawi, a combination of TA visits, a resident advisor, and a local consultant were used, resulting in a higher-cost survey process and package. The Steering Committee for the MDHS makes major decisions on survey content and utilization. The NSO is a major stakeholder and recipient of on-the-job training TA. ICF has used African consultants to assist with survey implementation in Malawi and this was reported to have worked well. This was seen as positive since the consultant was familiar with the country and its culture. However, several respondents noted that were limitations on the availability of African experts for short-term assignments. For the 2010 MDHS, on-the-job TA consisted of help from ICF country managers and other technical experts from Calverton in survey planning, monitoring of field work, and data processing. After data entry and initial consistency checks were completed, all remaining data processing and tabulation work for the Final Report was done by ICF in the U.S. with limited involvement of NSO counterparts. Another interviewee noted that a regional office would be useful given the number of surveys always in process throughout the region; such an office could also be used to provide capacity-building support for the dissemination and further analysis of DHS results. Some suggested that longer-term contracts would allow consultants to work more extensively on behalf of DHS. 3. Effectiveness of Capacity Building (Task 2) With respect to capacity building at the NSO, areas of ongoing need include sampling, data processing, and further analysis of data. The NSO would like TA visits to be longer (or be done by a resident advisor) so that staff can learn more and more effectively strengthen skills. Several respondents remarked that the NSO already has relatively strong capacity in interviewer training and fieldwork management (data collection) so less TA is needed there. It was observed that the NSO benefitted from TA in questionnaire design, data collection, and data entry/cleaning, but the NSO could still benefit from more workshops on sampling and data processing especially if NSO and ICF staff can work together. A sampling workshop was conducted for the 2010 survey, but the NSO did not actually draw the sample for the survey. One interviewee commented that capacity building is difficult due to staff turnover, lack of specialized skills, and the brain drain of well-trained staff. Another remarked that challenges for capacity-building include low staff retention and tension between capacity-building and the speed at which surveys must be completed. It was noted that emphasis should be placed on spending more time and effort to encourage use of DHS data at the local/district level. This might require a long-term effort and perhaps other stakeholders could take this on. EXTERNAL EVALUATION OF MEASURE DHS 76 Another respondent reported that the DHS Final Report is a quality product, but much of the report writing was done by ICF. It was recommended that a more strategic approach to capacity-building be undertaken with capacity benchmarks identified during the survey design process. Further capacity building is needed on data dissemination, further analysis, and data utilization. Most of ICF’s capacity building activities have been NSO focused and has sparked a demand in other ministries and civil society organizations for capacity building in data use and analysis. Numerous respondents noted that further analysis of the DHS is a priority. The MOH and NAC have met separately to look at ways of stimulating further data use and have developed a paper showing data gaps that require more investigation. Many observers felt that ICF is doing great capacity building work in Malawi, but more needs to be done in promoting the use and further analysis of data. This activity largely depends on the levels of funding available to support it. In Malawi, there have been two main further analysis efforts: • A Data Use Workshop organized by the MOH to identify data gaps in the MDHS Final Report that reached agreement on priority research questions for further analysis • A DHS Further Analysis Workshop in Zomba conducted in March 2012 These workshops focused on understanding and using DHS data and were considered helpful. Debate sometimes arose as to who was invited to the workshops and who was omitted, and was the case in Malawi. Respondents noted that there should be more dissemination of MDHS data at the district level. Health program planning happens at the district level and more capacity-building is needed to understand and appropriately use the DHS data. There is also a need to disseminate DHS data to parliamentarians and especially to the Parliamentarian Committee for Health. Discussions about the timing of DHS surveys should be coordinated with the Ministry of Economic Planning and Development so that data can be made available government-wide when needed. TA planning should begin with an assessment of financial and technical competencies to identify gaps requiring TA. Several respondents also stated that there needs to be more involvement of local stakeholders throughout the whole survey process. The Technical Working Group should be meeting more regularly to provide technical input to the NSO. Local stakeholders also wanted to be more involved in the tabulation and analysis phases of the survey. UNICEF has been supporting the Malawi Social and Economic Database (MASEDA) since 2001. It is a compilation of databases from all social and economic surveys undertaken in Malawi. The 2010 MDHS and previous DHS surveys are included. All these surveys are downloadable from the MASEDA website but rural districts lack sufficient bandwidth to download these files over the internet; thus, UNICEF is providing the database on CDs, TA on data utilization, and updated computer equipment to several districts each year so they can use MASEDA data for M&E. Capacity-building is needed on data collection, data processing, and data utilization at the district level. MDHS data dissemination is also needed at the district level. EXTERNAL EVALUATION OF MEASURE DHS 77 Cambodia Case Study Background The 2010 Cambodia Demographic and Health Survey is the most recent DHS to be completed in Cambodia. The survey included all standard DHS indicators plus additional modules for maternal mortality, abortion, injury and accidents, and use of health services. Biomarkers were collected for anemia in children under five and women of reproductive age. The national sample included 16,500 households consisting of 17,300 women aged 15-49 and 7,000 men aged 15-59. Implementers included the National Institute of Statistics (NIS), the Ministry of Planning, and the Ministry of Health. Funding was provided by UNFPA, USAID, UNICEF, JICA and the Second Health Sector Support Program (HSSP2), supported by the World Bank and a “pool” of bilateral donors. Evaluator Description 1. Meeting Stakeholder Needs (Task 1) In Cambodia, DHS data are widely used and trusted. DHS findings feature prominently in the country’s five year development plan as well as health sector budgets and program strategies developed by the Ministry of Health. Many of the 120 indicators used in the current national health plan for the country come from the DHS. The maternal and early child health units within the MOH use the DHS for detailed annual planning down to the district levels. With assistance from UNAIDS and UNICEF, DHS results have also informed nutrition and HIV programming. The government (which has a five year planning cycle) delayed their last five year plan by a year in order to incorporate current DHS results, causing some difficulties in allocating national and local area budgets on schedule. There is a widespread view in Cambodia that senior and middle-level government officials are not fully conversant with the DHS and the importance of its main findings. One interviewee observed that there is no local resource person (national or expatriate) who can address survey results with authority and this sometimes results in confused interpretations of key indicators such as HIV prevalence, the maternal mortality ratio, and child mortality. In addition, almost all informants said that the highest priority was the development of greater analysis capacity, illustrating that the country is not making as much use of the DHS as it could. In Cambodia, USAID plays a minor role in determining the final content of the DHS. The Ministry of Health is responsible for convening a Steering Committee for the survey and the National Institute of Statistics is responsible for convening a Technical Committee. These bodies review the preliminary core questionnaire submitted by ICF and make recommendations on what alterations to the proposed survey design may be needed. A large workshop of stakeholders provided another avenue for gaining broad input. The process has worked well in letting everyone feel they have an opportunity to influence decisions on the content of the survey. One respondent noted that in Cambodia, “everyone owns the CDHS." Ultimately it is the Ministry of Health, working with the National Statistical Office and ICF, which decides on the final design of the questionnaire and issues pertaining to technical and operational feasibility. ICF collaborates to make decisions regarding technical issues and questionnaire design and content. 2. Effectiveness of Technical Assistance (TA) Most respondents considered the quality of TA from ICF to be of a high standard. It was generally acknowledged that external TA will still be necessary for some time to come, with one respondent estimating at least another 10-15 years. For the next DHS round, more careful consideration needs to be given to aspects of survey implementation that can be handled by the NIS on their own and elements of survey operations that will continue to benefit from external TA from ICF. One respondent commented that the EXTERNAL EVALUATION OF MEASURE DHS 78 government could likely do the CDHS on its own, but they will likely require TA for the incorporation of new survey methods and modules. It was judged important that the CDHS remain consistent with international standards and practices. USAID usually accepts the basic package of TA recommended by ICF. There is little systematic investigation of TA needs or rationales provided for the level of TA support. Several informants stressed the benefit of planning TA inputs well in advance of the next survey. This planning process should involve major stakeholders with oversight responsibilities for the DHS. Surveys and censuses in the NIS typically have international consultants who stay for long periods of time and explicitly build training activities into their agendas. The hiring of a local ICF consultant to work at the NIS proved to be a technically efficient and cheaper option than hiring an expatriate adviser. Since there was not a scope of work to hire a fulltime survey coordinator from within the government, the use of a local consultant proved to be a good option especially given the adviser’s previous experience managing other surveys. 3. Effectiveness of Capacity Building (Task 2) There was a widespread view among respondents that the capacity of NIS to conduct surveys has improved, not just due to the DHS but also because of its involvement in other surveys and census operations. One respondent noted that ICF was impressed with the growing capacity of NIS, especially in data processing. A major problem faced by the 2010 CDHS was a ruling by the government that barred payment of incentives for work on projects such as the DHS; this constraint proved difficult for some staff to remain fully committed to the survey. Numerous respondents recommended that capacity building for the CDHS should be given higher priority through workshops, on-the-job training, and opportunities for international technical exchanges and collaborative analysis/research. One interviewee noted that JICA does more training at NSI than ICF and they do more to assess people's need for training and more effectively sets expectations for whether capacity goals have been met. JICA is also conducting training programs in the provinces. Most interviewees strongly believe that capacity building needs still exist. Major capacity building priorities identified by respondents include: (1) training in CSPro and (2) data analysis and report writing. With respect to data processing training, one respondent commented that this instruction must be made specific to the CDHS with NIS staff actually doing the work and then having their efforts cross-checked by ICF and other expert staff. Another observer felt that data analysis should be a major priority since it is uncomfortable for local people to not be able to talk about and use DHS data. Report writing was also thought to be very weak in Cambodia and needs to be improved; future capacity building in this area needs to focus on producing tables and written descriptions of them. The long-term goal of these efforts should allow for the production of final DHS tabulations and country reports in Cambodia rather than having much of this work done at ICF headquarters. One observer noted that there is low data analysis capacity in all branches of government. This capacity is somewhat stronger in the NIS and that is where skills should be developed. This respondent noted that it might be possible to engage more productively with local universities and research institutions like the Cambodian Development Resources Institute in building capacity, but there are lots of obstacles including staff turnover. It was also recommended that training be provided only to people with some foundation in statistics. In addition, more attention needs to be given to building greater expertise in specialized fields of public health such as reproductive health, child health, and nutrition. Several informants stated that the first in-country analysis/research activity ever undertaken for the CDHS were two recently conducted data user workshops for the 2010 CDHS. While this may not be completely EXTERNAL EVALUATION OF MEASURE DHS 79 true, it is an important perception. UNFPA, which supports the NIS, hosted training on use of CDHS information in which USAID covered ICF costs. It was also noted that more training will be needed after these workshops. One respondent hoped it will be possible to use the “deep analysis skills” gained from DHS workshops to analyze the country’s commune data base which is just being put on the internet––it had been paper based before. It was observed that household survey capacity building had only just begun in Cambodia. It was also noted that capacity building goals should be identified well ahead of the survey, discussed with all stakeholders, and monitored with prescribed milestones and dissemination activities should be planned at the same time. It was thought that making DHS results more widely available will help increase the number of people and sectors who can interpret and effectively use CDHS indicators and tables. Bangladesh Case Study Background The 2011 Bangladesh DHS (BDHS) was conducted in 2011. The sampling frame included 17,141 households consisting of 17,842 women aged 12 to 49 and, 3,997 men aged 15-54. The biomarkers collected included anemia, fasting glucose, iodine salt, and micronutrients. Preliminary results were released in April 2012. The previous BDHS survey was completed in 2007. The 2007 and 2011 surveys were conducted under the authority of the National Institute for Population Research and Training (NIPORT) of the Ministry of Health and Family Welfare (MOHFW) and implemented by Mitra and Associates, a Bangladeshi private-sector research firm based in Dhaka. Evaluator Description 1. Meeting Stakeholder Needs (Task 1) The DHS is extremely well regarded in Bangladesh and considered to be the most credible source of national demographic and health data in the country. Several respondents noted that having an international firm like ICF to ensure the integrity of the BDHS greatly enhances its credibility among potential stakeholders. More individuals and institutions are using BDHS data than ever before and BDHS findings appear regularly in the media and are in greater public view than ever before. Several respondents noted that the BDHS has become the “gold standard” for demographic and health information in Bangladesh. Other data sources commonly used in the country include the sample vital registration system implemented by the Bangladesh Bureau of Statistics, UNICEF MICS surveys, household expenditure surveys, and Matlab data from ICDDR/B. DHS data are used to monitor 17 of 41 key indicators in the Health Nutrition and Population Sector Program of the Ministry of Health. The DHS is also used to track 24 of the Health SWAP indicators that address the status of health needs in the country. As part of the dissemination activities in Bangladesh undertaken for the 2007 DHS, seven presentations of DHS findings were organized for regional government health officials and NGO representatives. In addition to these sessions, 30 Upazilla (county) events were held for local officials. DHS findings have also been used to prepare thematic policy briefs for local area health program managers and women members of parliament (MPs). Two rounds of training for journalists on DHS methods and findings (the second round conducted in collaboration with Voice of America) have also been implemented; the first of these workshops was conducted in 2009 for 18 national and 20 local journalists with backgrounds in health reporting. Numerous respondents noted that the DHS questionnaire is well done and satisfies many essential health information needs, but it does not meet all of them. Missing topics include questions on aging, migration, and EXTERNAL EVALUATION OF MEASURE DHS 80 climate/environment. Some also claimed the Final Report is too technical for some audiences and not user￾friendly. Numerous respondents noted that policy briefs are needed to make DHS data and findings more accessible to policymakers, NGOs, and government officials. There is also a need to find better ways to communicate key findings to health program managers at the district level. Several respondents believe that the high degree of standardization in the DHS core questionnaire has frustrated efforts to make the survey instrument more relevant to local program needs. The DHS could provide greater clarity on the problems clients face in acquiring essential commodities and health services and identify characteristics of health seeking behavior and treatment patterns. Greater use of interesting and helpful ways to make information available on the Internet for division and district health workers would also be useful. One respondent also noted that the 2007 data seemed outdated when the DHS national seminar was held in 2010. It was concluded that dissemination should be done sooner after data are collected from the field. The Social Marketing Company (SMC) coordinates commercial marketing of family planning and other health products across the country and would be interested in having the DHS collect more information on nutrition and food security. A new micro-nutrient powder (MoniMix) being marketed in Bangladesh has considerable potential for further reducing levels of malnutrition (primarily iron deficiency anemia in children). Oral saline use for the treatment of diarrhea in both children and adults has also not been studied following the introduction of new ORS formulations. In Bangladesh, the Technical Review Committee (TRC) is the main steering committee for the DHS. The TRC is a large consultative group consisting of senior government officials, NGOs, major donors, international organizations, and other stakeholders engaged with the DHS. The TRC is most active in the design of the core questionnaire, review of preliminary and final reports, and dissemination efforts. A smaller Technical Working Group comprised mainly of experts from USAID/Bangladesh, NIPORT, and ICDDR/B monitor and advises on all aspects of survey implementation. NIPORT, in consultation with the USAID Mission, selects people to serve on both groups. The TWG for the 2011 BDHS is largely made up of individuals from NIPORT, ICDDR/B, and the USAID Mission. The Mission tends to be more actively involved in these negotiations compared to other partnering countries owing to the presence of an experienced demographer/health statistician based at USAID’s Office of Population and Health (OPH). 2. Effectiveness of Technical Assistance (Task 1) Mitra and Associates was able to conduct the 2011 BDHS without too much TA from ICF. ICF experts made six TA visits related to sampling, biomarker training, data processing, and survey coordination. These experts did not necessarily visit NIPORT when they were in Dhaka. A major TA contribution for the 2011 BDHS was a south-to-south collaboration on glucose field-testing provided by a Bangla speaking consultant from Nepal. Data processing assistance was provided by ICF over a two-week period in 2010; however, several respondents noted that Mitra’s data processing staff would have preferred more hands-on involvement with the TA supplied by ICF and more mentoring in the use of CS Pro. Interviewees also noted that ICF provided highly effective and valued assistance in data dissemination, especially in training journalists to use DHS findings. One respondent remarked that Bangladesh would need technical assistance in all aspects of household survey operations for some time to come; this is especially true given the growing size and complexity of the DHS in Bangladesh. It was noted that reducing or ending funding for ICF technical assistance before other data systems were adequately developed would be a major setback for the country. There was broad consensus that Mitra and Associates generally did a good job in training interviewers and ensuring high standards in collecting data. ICF’s technical collaboration in the timely production of tabulations and preliminary and final country reports for the DHS surveys was viewed as well-timed and EXTERNAL EVALUATION OF MEASURE DHS 81 highly effective. It was recommended that future BDHS survey contracts include a plan to document what TA is needed and how it could be best to further develop the capacity of local implementing partners. 3. Effectiveness of Capacity Building (Task 2) Over the years in Bangladesh, there has been good progress in building the capacity of private sector firms to conduct household-based surveys. SN Mitra, the Director of Mitra and Associates (the LIA for the DHS), credited early technical (mentoring) support from the USAID/Mission for much of the success of his firm and made special mention of Carol Carpenter-Yaman’s early support. Mitra and Associates currently do some capacity building of their own. For example, staff members have organized seminars on data collection at the BRAC School of Public Health and another local university in Dhaka. They spent their own resources on this activity as their unique contribution to capacity building. Several LIA respondents noted that the BDHS was becoming increasingly large and complex and that there were increasing concerns about its long-term viability. The introduction of biomarkers for fasting glucose (diabetes) measurement and nutritional intake placed a considerable burden on the 2011 BDHS survey instrument, although this information appears to have been successfully collected. Many felt there were limits to the range of information a household-based survey could collect and that in the case of Bangladesh, these limits were likely being approached. Data processing staffs at Mitra and Associates and NIPORT are interested in becoming more experienced in the use of CS Pro. They consider themselves to be largely self-taught in the use of this data entry/editing software and would like to become more conversant in its use. DHS data analysis is not straightforward, posing a barrier to further analysis efforts using electronic data files. Further analysis workshops would be useful to familiarize participants with using data files for further analysis. It was noted that extended analysis of the 2007 BDHS “never came together,” although some university students have been using the 2007 data file for course work and research projects. Unlike the BDHS rounds during the 1990s (and the predecessor CPS surveys of the 1980s), no formal secondary analysis activity was organized. One interviewee commented that the training of educators and journalists in using DHS data was fine, but the BDHS should be more widely exploited as a research resource. One respondent emphasized that more work is needed to make DHS data accessible to in-country stakeholders and to overcome barriers to more widespread utilization of the data for program planning and evaluation, especially by other disciplines. Capacity-building and data dissemination should also go out to all District-level health managers; part of this effort would require additional policy reports that more clearly draw out the program and policy implications of DHS findings. There was a general consensus among respondents that more in-country training on household survey procedures should be conducted and greater efforts should be made to utilize local expertise in providing technical assistance for future surveys. There was a desire to see more of a “bottom-up approach" for TA rather than relying on external expertise. DHS Final Reports are now mostly written by local authors, an indicator of growing local capacity to implement the DHS. However, several observers said that having a wide array of local stakeholders involved in preparing chapters for the final country report was not a very efficient approach. The quality of the first drafts were uneven and required considerable re-writing by ICF staff; having the writing done at ICF headquarters with senior staff from Mitra and Associates and NIPORT collaborating in the work (as was the practice for BDHS operations prior to 2007) was actually more productive and cost effective. However, the in-country writing workshop approach will likely be the modality for the 2011 BDHS and future surveys as well. EXTERNAL EVALUATION OF MEASURE DHS 82 It was observed that the most immediate challenge for capacity building is the effective engagement of health professionals and program implementers in the use of DHS data. There is also considerable potential to increase the use of on-line instructional materials that support the use and interpretation of DHS findings, especially if more training tools could be made available in Bengali as well as English. It was recommended that ICF should consider promoting greater use of e-learning tools and video training. DHS dissemination and capacity building efforts have largely been at the country's initiative. Future capacity building efforts should focus on building-up teams of people in-country who are experts in questionnaire design, sampling, data processing, data collection, and analysis. Creating and strengthening partnerships with research institutions, NIPORT, and the country’s extensive and rapidly growing university community would also be important. There is also a continuing need to further build the capacity of government statistical agencies (e.g., NIPORT, the MIS Unit of the MOHFW, and the Bangladesh Bureau of Statistics) and private companies to implement quality surveys. One respondent commented that there is little doubt that Mitra and Associates could do the DHS on their own, but they also asked, “Would it have the same credibility without ICF involvement?” Tanzania Case Study Background Tanzania has had three MEASURE DHS surveys since 2008. They include the 2009-10 Tanzania DHS, the 2010 Micronutrient Survey, the 2007-08 Tanzania HIV/AIDS Malaria Indicator Survey (THMIS), and the 2006 Services Provision Assessment (SPA). The DHS included the collection of several biomarkers: anemia, Vitamin A, iron deficiency, and urinary iodine levels. Samples for the DHS included 10,000 households consisting of 14,400 women and 2,600 men. The 2007-08 Tanzania HIV/AIDS Malaria Indicator Survey (THMIS) integrated HIV and malaria questions and biomarkers including anemia as well as HIV and malaria prevalence. The sampling frame included 8,500 households consisting of 9,340 women and 7,000 men. The 2006 Services Provision Assessment (SPA) was conducted in 611 facilities. Evaluator Description 1. Meeting Stakeholder Needs (Task 1) The TDHS, THMIS, and Micronutrient Survey are the main sources of demographic and health information in Tanzania. The data are regularly used by a wide variety of stakeholders including the national HIV/AIDS and malaria control programs. However, there was little mention of the 2006 SPA; this resource appears much underutilized and is possibly attributable to the length and highly detailed nature of the SPA. DFID provided funding for the in-country costs of the 2010 TDHS while USAID covered the costs of ICF. The NBS noted that the DHS has become far richer in content, but they also worry that the surveys are growing too large and average interview times have become excessively long. Another respondent noted that the THMIS may have gotten too big and that there is too much time between surveys for the instrument to be useful in monitoring malaria prevalence and prevention and treatment outcomes. One respondent would preferable to see the THMIS spilt into two stand-alone surveys, with the malaria component implemented every two years during peak malaria transmission seasons. However, the integrated survey format seems to be working "well enough," and there are obvious cost savings for using dried blood spots for multiple biomarker tests. No respondents were critical of the surveys for omitting important health topics; however, respondents noted there was still useful information that could be collected in future surveys. Some issues that were identified by respondents for future inclusion include additional reproductive health issues; TB, which is currently not EXTERNAL EVALUATION OF MEASURE DHS 83 being addressed in household surveys; and the field of non-communicable disease, which remains largely untouched. Currently, Tanzania also has little information on the diagnosis and treatment of sexually transmitted diseases, post-abortion care, and referral and treatment patterns (including access to emergency obstetric care) for complicated deliveries. Some respondents pointed out that too much reliance was being placed on household surveys to provide Tanzania's core health information. It was noted that Tanzania’s decentralized health system requires a functional health management information system (HMIS) to generate the district-level information needed to function. The current donor response to health information needs was characterized as confused, uncoordinated, and counter-productive. It was argued that donors are developing their own project-specific surveys and MIS operations to report on the outcomes of individual projects rather than investing in a viable national HMIS system. In addition, vertical disease control programs (e.g., HIV/AIDS, malaria, and TB) have their own reporting systems independent of the HMIS. In Tanzania, there has been good stakeholder involvement in the development of the 2010 TDHS, 2007 and 2012 THMIS, and 2010 Micronutrient Surveys. Survey content in Tanzania is agreed upon through a consultative process coordinated by the National Bureau of Statistics. Among the major stakeholders are the Ministry of Health and Social Welfare, Ministry of Education, and Ministry of Social Affairs and Community Development. International organizations and major donors in the health sector also participate in these discussions. Once these stakeholder consultations are complete, ICF “steps forward to put everything together,” including the development of a final budget for all phases of the survey. The Technical Committee then oversees the implementation of the survey. There was considerable satisfaction among interviewees from the National Bureau of Statistics, Ministry of Health and Social Welfare, and other stakeholders with the way this process has functioned over numerous DHS rounds. The Malaria Control Board was a major stakeholder in the development of the THMIS survey and questionnaire design. They have also played a consultative role in finalizing biomarker algorithms used in testing for the four main malaria strains in Tanzania (predominantly plasmodium falciparum). A recent country report on progress in combating malaria was sponsored by Roll Back Malaria and prepared in collaboration with the NMCB, MOHSW, and WHO. The report was notable for its wide stakeholder collaboration that included individuals from PMI, CDC, Ifakara Health Institute, Tulane University, Johns Hopkins University, and Commit – the Communications and Malaria Initiative in Tanzania. The report drew widely on TDHS and THMIS information between 1999 and 2010 in areas such as malaria prevalence, use of insecticide-treated nets (ITNs), artemisin-based combination treatment for children (ACT), and intermittent anti-malarial drug treatment for pregnant women (IPTP therapy). 2. Effectiveness of Technical Assistance (Task 1) ICF’s TA was generally judged to have been effective. It was appropriately designed for local needs and timely delivered. Support for questionnaire design, sampling, biomarker generation, data processing, analysis, report writing, and the seven zonal dissemination seminars were most readily cited as areas effectively addressed by ICF TA. The technical assistance provided to NBS focused on questionnaire development, sampling, data processing, and the dissemination of results. The NBS would appreciate having additional assistance on sampling in future years given the growing complexity of survey designs. Additional instruction in CS Pro and indicator instruction for data processing staff would also be useful. The MOH would also like to have their staff participate in more data tabulation and analysis programs for the TDHS and THMIS. It was noted that MOH staff need more "hands-on" experience in handling DHS data files as well as further instruction in appropriate analysis and research methods. NBS staff also singled out ICF assistance in preparing testing algorithms for the HIV, malaria, and micro-nutrient biomarkers. EXTERNAL EVALUATION OF MEASURE DHS 84 The NBS would also like to see more utilization of DHS materials in Tanzania. They noted that data analysis and report writing were still poorly developed skills in Tanzania. Appreciative mention was also made concerning efforts to disseminate 2010 TDHS results to stakeholders in the seven zones of Tanzania; to date, three regional dissemination events have been held. Tanzania will continue to have TA and capacity building needs in running household surveys in the future; sampling, data processing, analysis, and report writing are areas that were often mentioned. The NBS would also like to see the introduction of automated data entry devices in future surveys to take pressures off field editors and speed up data processing times. 3. Effectiveness of Capacity Building (Task 2) Capacity at the NBS has been built over the years through its association with ICF, but there continues to be capacity constraints that require additional attention most notably in sampling, data processing, and analysis/report writing. Many respondents mentioned that the NBS is now far more skilled in conducting household-based surveys, but continued association with ICF will continue to be important for future survey rounds. It is anticipated that the need for external capacity building efforts will diminish in future surveys; however, several respondents worried about the sustainability of the current technical proficiency of NBS since several senior officials will soon be retiring and the supply of new junior professionals in demography and statistics from Tanzania’s university system is considered inadequate to meet future needs. Given data needs for evaluating short-term change in the incidences of malaria, several respondents noted that the THMIS is not an ideal instrument; having to wait 5-6 years between surveys frustrates the need for more current information. Confidence intervals for some key estimates were unsettlingly large and district￾level data (the locus of malaria control interventions) were not available from the THMIS. As far as the malaria community is concerned, it is time to wean Tanzania off its heavy reliance on household-based survey information. Instead, greater technical support and resources should go into strengthening malaria diagnostic, prevention, and treatment information in the HMIS even if it is done at the expense of some behavioral information. It was thought that the need for more current malaria prevalence information outweighed other considerations. The Tanzania AIDS Commission would like to see greater efforts made to link HIV findings from the THMIS with other data sources, particularly facility-based data. It was noted that facility-based information should be able to provide greater detail on the availability of HIV/AIDS prevention, testing (VCT), and ARV treatment services at MOHSW clinic sites. In general, the THMIS was judged to be working fine as an integrated survey instrument combining HIV and malaria components. It is a valuable source of information on HIV knowledge, risk behavior, and stigma. Nonresponse (refusal) rates in the 2007 THMIS have been quite low, raising confidence in THMIS data quality. The TA provided by ICF in the development of the micronutrient questionnaire and biomarker-testing protocols has greatly increased the ability of the Tanzania Food and Nutrition Center (TFNC) and the NBS to undertake micronutrient surveys. There will be considerable scope for widening the coverage of nutritional information in the future as other health conditions (including chronic disease) receive greater attention from nutritionists. EXTERNAL EVALUATION OF MEASURE DHS 85 I. Tabulations from the E-survey TABLE 1: WHAT TYPE OF DHS SURVEYS WERE MOST RECENTLY COMPLETED IN YOUR COUNTRY BETWEEN 2008 AND 2011? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Standard Demographic and Health Survey 59.1% 66.7% 60.7% 17 AIDS Indicator Survey (AIS) 0.0% 0.0% 0.0% 0 Service Provision Assessment (SPA) Survey 9.1% 0.0% 7.1% 2 Malaria Indicator Survey (MIS) 13.6% 33.3% 17.9% 5 Key Indicator Survey (KIS) for Small Areas 0.0% 0.0% 0.0% 0 Interim DHS 9.1% 0.0% 7.1% 2 Knowledge, Attitudes and Practices (KAP) 0.0% 0.0% 0.0% 0 Other 9.1% 0.0% 7.1% 2 Other (Please specify) 8 3 11 Answered Question 22 6 28 Skipped Questions 1 0 1 TABLE 2: OVERALL, DID THESE SURVEYS MEET YOUR COUNTRY’S NEEDS FOR INFORMATION? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Yes, extremely well 85.0% 50.0% 76.9% 20 Yes, for some needs 15.0% 50.0% 23.1% 6 No, it is not very useful 0.0% 0.0% 0.0% 0 Answered Question 20 6 26 Skipped Questions 3 0 3 EXTERNAL EVALUATION OF MEASURE DHS 86 TABLE 3: PLEASE DESCRIBE ANY SHORTCOMING THAT REDUCED THE USEFULNESS OF THE SURVEYS - (OPEN-ENDED RESPONSES) 1. The survey is ongoing. Initial data collection has been completed. There will be some follow-up data collection. Test results, data entry, data analysis, etc., still remain to be done. 2. Not done per county, only per region. 3. There is no issue with the data per se, just the long time that it has taken to get the results (for the 2010 DHS). Data collection took longer than expected, and now a year later we still do not have the final report. 4. Not much malaria info as I understand it. Results not yet shared except for summary, 5. The analyses of the data need to be more policy-oriented. 6. It would have been useful to gather unit cost information-- particularly for the private sector. 7. The DHS 2011-2012 is still collecting information in the field we will have the final report by the end of 2012 or beginning 2013. We had some coordination meetings with organizations of the health sector as the ministry of health, donors and health organizations to receive input of those topics included in the DHS and that compared from the previous DHS 2005- 2006 should be adjusted to make them more useful. 8. Due to changes in the generic questionnaire some of the data are not comparable with data from previous DHS, which makes it impossible to track changes over time. 9. Unwillingness of MACRO to provide preliminary tables in accordance with programming needs/schedules. 10. The study was just completed. In-depth analysis by in country stakeholders has not started. 11. The results are not yet available but the country counts on it for its strategic programming. 12. Delays in conducting and releasing results of the AIDS Indicator Survey. 13. Excessive delay between survey and final report reduces the ability to use the data for action. 14. Capacity of in-country partners should be developed. Answered Questions 15 Skipped Questions 14 TABLE 4: HOW DID YOU OR YOUR OFFICE USE THE RESULTS OF THE MOST RECENT DHS? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Reporting requirements 66.7% 80.0% 76.9% 20 Programming programs/projects 83.3% 90.0% 88.5% 23 Policy dialogue 66.7% 85.0% 80.3% 21 M&E of programs/projects 83.3% 95.0% 92.3% 24 Other – please specify 33.3% 15.0% 19.2% 5 EXTERNAL EVALUATION OF MEASURE DHS 87 Answered Questions 20 6 26 Skipped Questions 3 0 3 TABLE 5: WHAT OUTPUTS FROM THE MOST RECENT DHS SURVEYS DO YOU OR YOUR COUNTRY OFFICE USE REGULARLY? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Complete survey data file 42.9.% 16.7% 37.0% 10 Final country report 90.5% 66.7% 85.2% 23 Summary or Key Findings report 76.2% 83.3% 77.8% 21 Dissemination posters 19.0% 16.7% 18.5% 5 Further analysis reports 42.9% 0.0% 33.3% 9 STAT Compiler on the DHS website 33.3% 0.0% 25.9% 7 Other – please specify 9.5% 0.0% 7.4% 7 Answered Question 21 6 27 Skipped Questions 2 0 2 TABLE 6: WHAT RESULTS FROM THE MOST RECENT DHS WERE MOST ESSENTIAL FOR YOUR PROGRAM EFFORTS? 1. IMR, CMR, MMR, TFR, and province specific data for malnutrition and service coverage. 2. Survey not completed yet. 3. Malaria data. 4. ITN coverage and use, IPTp coverage, parasitemia. 5. Indicators for malaria line ITN coverage, IPTp coverage, and health seeking behavior. Also accessibility to radio and TV for behavior change and communication interventions. 6. Contraceptive use, HIV seroprevalence, infant and child health. 7. Under 5 year mortality rate. 8. Infant and child health and malaria indicator information. 9. Infant and child health; malaria/anemia prevalence, malaria prevention usage and access. 10. Child malnutrition, hepatitis C prevalence, contraceptive prevalence rate, TFR, infant and child mortality. 11. Chronic Malnutrition, Under five mortality rate, Infant mortality rate, maternal mortality, EPI coverage. 12. MMR, HIV prevalence, health indicators by quintile. 13. We expect that this DHS will allow you to see the impact of the implementation of maternal and child health indicators (i.e. Total Fertility Rate, Use of modern contraceptives, Neonatal mortality rate, infant mortality rate, rate of skilled birth attendance, nutrition related percentages as percentage of children under 5 with chronic, acute and global malnutrition, anemia related percentages for women and children and HIV. EXTERNAL EVALUATION OF MEASURE DHS 88 14. Data on MCH, FP/RH, health service utilization, fertility and abortions, women's empowerment. 15. All data 16. HIV/AIDS, OVC, Fertility. 17. Fertility rate, contraceptive use, infant and child health, EPI coverage, and HIV seroprevalence. 18. Total Fertility Rate, Contraceptive Prevalence Rate, Child Mortality Rates, Nutritional Indicators. 19. Infant and child health, EPI coverage, contraceptive use, HIV seroprevalence. 20. Malaria parasite prevalence, CPR, infant and child health. 21. CPR, TFR, Nutritional Status, Malaria data. 22. All of the above. The survey is broadly useful and accepted as the best set of impact and program data in my country. HIV seroprevalence is done in a different survey, however. 23. 1. Fertility and contraceptive rate; 2. Skilled birth attendance; 3. Stunting and underweight; 4. Knowledge of contraceptive methods; 5. Trends in infant and child mortality; 6. Incidence and treatment of diarrhea and pneumonia; 7. Prevalence of anemia in children and anemia. 24. All (except HIV seroprevalence as not collected in our DHS). 25. New mortality rates. deeper understanding of factors influencing use of ANC. 26. HIV seroprevalence. TABLE 7: WHO ARE THE MAJOR STAKEHOLDERS IN THE COUNTRY WHO WERE INVOLVED IN DEVELOPING THE MOST RECENT DHS IN YOUR COUNTRY? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Local NGOs 0 0 0 0 World Bank 1 0 1 1 UNFPA 9 0 9 9 UNICEF 11 2 13 13 Ministry / Department of Health 19 5 24 24 Other host government ministries (specify) 12 2 14 14 USAID Mission 18 5 23 23 PMI 7 4 11 11 CDC 4 4 8 8 Regional universities and research institutes 2 1 3 3 International NGOs 3 0 3 3 Other (please specify) 9 3 12 12 Answered Question 21 6 27 Skipped Questions 2 0 2 EXTERNAL EVALUATION OF MEASURE DHS 89 TABLE 8: OVERALL, WAS MACRO RESPONSIVE TO THE INFORMATION NEEDS OF THESE STAKEHOLDERS? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Yes, very responsive 52.4% 16.7% 44.4% 12 Yes, responsive within technical or budgetary constraints 42.9% 33.3% 40.7% 11 No, not responsive 4.8% 0.0% 3.7% 1 Don’t know 0.0% 50,0% 11.1% 3 Answered Question 21 6 27 Skipped Questions 2 0 2 TABLE 9: TO THE BEST OF YOUR KNOWLEDGE, WHO WERE THE MAJOR USERS OF THE MOST RECENT DHS IN YOUR COUNTRY? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Ministry / Department of Health 19 5 24 24 Local Universities 8 0 8 8 International NGOs 13 4 17 17 Local NGOs 7 1 8 8 UNICEF 18 5 23 23 UNFPA 17 4 21 21 World Bank 13 4 17 17 USAID 19 5 24 24 PMI 8 5 13 13 CDC 9 5 14 14 EXTERNAL EVALUATION OF MEASURE DHS 90 Other (please specify) 8 0 8 8 Answered Question 20 6 26 Skipped Questions 3 0 3 TABLE 10: HOW WOULD YOU DESCRIBE ICF INTERNATIONAL’S LEVEL OF TECHNICAL ASSISTANCE (TA) EFFORT DURING THE RECENT DHS SURVEYS? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Extensive TA during all phases of survey operations 70.0% 16.7% 57.7% 15 Moderate TA provided during selected phases of survey operations 25.0% 50.0% 30.8% 8 Limited TA provided by ICFI 0.0% 0.0% 0.0% 0 No TA provided by ICFI 0.0& 0.0% 0.0% 0 Don't know 5.0% 33.3% 11.5% 3 Answered Question 20 6 26 Skipped Questions 3 0 3 TABLE 11: WHAT HAVE BEEN THE MAJOR SOURCES OF TA DURING THE IMPLEMENTATION OF YOUR LAST DHS? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count ICFI staff visiting from Calverton 18 44 22 22 Resident technical advisers 2 1 3 3 National professionals 4 2 6 6 EXTERNAL EVALUATION OF MEASURE DHS 91 South-to-south technical consultants 3 0 3 3 USAID Mission staff 7 2 9 9 Macro emails or other virtual TA 13 4 17 17 Other 0 1 1 1 Answered Question 20 5 25 Skipped Questions 3 1 4 TABLE 12: IF YOUR COUNTRY UNDERTAKES ANOTHER DHS, WOULD YOU PREFER THAT TA BE PROVIDED IN A DIFFERENT MANNER? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Yes 55.0% 50.0% 54.2% 11 No 45.9% 50.0% 45.8% 13 If yes, please specify (see below) 16 Answered Questions 20 4 24 Skipped Questions 3 2 5 1. MACRO played a support role in this DHS. I would have them take full responsibility in the next DHS. 2. Despite admittedly difficult circumstances, with numerous delays and a brief civil war, we would like our TA partner to play a clearer, more proactive leadership role to ensure that planning, budgeting, and implementation are adequate, and to take a more proactive role in alerting us to potential difficulties. 3. Not necessarily how TA is provided, but we think that there is great local capacity to take more responsibility, if not all, for the overall survey. 4. But would like to see both UBOS and MOH take more responsibilities. 5. Build more local capacity to make TA needs limited. 6. To have more policy oriented analysis and presentations. 7. GoP (MEF and INEI) will define how, when and who will provide the TA. 8. More in-country capacity building on data entry, cleaning, and analysis and report writing. 9. Previously to this DHS we discuss with the ministry of health and the national institute that this will be the last DHS supported by the Mission and with this intention we agreed to ask ICFI to provide TA to develop to the national capacities so that next DHS's the country is ready to plan, conduct and perform the survey. 10. Very limited TA/oversight provided by one consultant from MACRO and more reliance on local implementing agency to plan and conduct the survey. 11. Resident Technical Adviser. 12. As national institutions develop capacity, Mentoring may be better than short term TA visits. EXTERNAL EVALUATION OF MEASURE DHS 92 13. Macro costs from USAID were not agreed on at the beginning which caused problem. 14. Macro should continue building capacity of local institutions such as Uganda Bureau of Statistics and local universities to take on the main roles of conducting these DHS surveys. 15. Use of the survey results at State and Local level needs to be increased. 16. We are exploring option from extensive Macro TA to limited Macro TA, and limited local institution to extensive role of local institution for upcoming surveys. TABLE 13: IN WHAT AREAS HAS THERE BEEN EFFORT TO BUILD DEMOGRAPHIC AND HEALTH SURVEY CAPACITY IN YOUR COUNTRY? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Improving the skills and self sufficiency of local implementing agencies and individual staff members to plan and conduct surveys and collect, process, and disseminate data 90.5% 66.7% 85.2% 23 Building capacity of labs and lab technicians to collect, process, and analyze biomarker data 47.6% 33.3% 44.4% 12 Increasing skills among multiple audiences to interpret and use DHS results for evidence￾based planning and monitoring 61.9% 33.3% 55.6% 15 Enhancing skills among researchers and academics to analyze DHS data sets 42.9% 50.0% 44.4% 12 No capacity building focus 4.8% 0.0% 3.7% 1 Don’t know, no opinion 0.0% 16.7% 3.7% 1 Other 4.8% 0.0% 3.7% 5 EXTERNAL EVALUATION OF MEASURE DHS 93 Answered Question 21 6 27 Skipped Questions 2 0 2 TABLE 14: HAVE YOU SEEN GROWTH IN CAPACITY IN THIS COUNTRY TO COLLECT, ANALYZE, OR USE POPULATION AND HEALTH SURVEY DATA IN THE PAST FOUR YEARS? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Yes, large growth 28.6% 16.7% 25.9% 7 Yes, some growth 38.1% 50.0% 40.7% 11 Yes, limited growth 23.8% 33.3% 25.9% 7 No growth 0.0% 0.0% 0.0% 0 Deterioration in capacity 0.0% 0.0% 0.0% 0 Don't know 9.5% 0.0% 7.4% 2 Please give specific examples where capacity has grown (see below) Answered Questions 21 6 27 Skipped Questions 2 0 2 1. Planning, organization, data collection and limited analytical skills. 2. DHS is been used by the MEF to evaluated the Program Budget by Results. The MOH is using the information to evaluate advances on the Millennium Objective Goals. 3. A lot of institutional users referenced in their studies or M&E presentations for analysis, planning and EXTERNAL EVALUATION OF MEASURE DHS 94 programming. 4. Level/intensity of TA needed from ICFI decreased between 2007 DHS and 2009 MIS - however closer TA needed for 2011 MIS and 2012 DHS due to change in personnel in key government institutions. 5. Being a survey that is implemented after 5 years, capacity building interventions remain a challenge because of turn over. 6. When programming health activities. 7. UBOS has been empowered to be able to lead all data collection efforts and preliminary analysis. Additional capacity building is required in complex analytical skills. 8. Data is used and quoted extensively for planning and monitoring. TABLE 15: OVERALL, HOW SATISFIED ARE YOU WITH ICFI’S CAPACITY BUILDING ACTIVITIES IN THE LAST FOUR YEARS? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Very satisfied, considerable improvement in capacity was observed. 28.6% 0.0% 22.2% 6 Satisfied - ICFI made contributions to capacity. 33.3% 16.7% 29.6% 8 Not satisfied -ICFI could have done more to build capacity. 19.0% 66.7% 29.6% 8 Very unsatisfied - ICFI did little or nothing to build capacity. 0.0% 0.0% 0.0 0 Don’t know - no opinion. 19.0% 16.7% 18.5% 5 Answered Question 21 6 27 Skipped Questions 2 0 2 TABLE 16: WERE FUNDS ALLOCATED AS SPECIFIC BUDGET ITEMS FOR CAPACITY BUILDING? USAID Field Missions PMI Advisers USAID + PMI Respondents Response Count Yes 57.1% 75.0% 60.0% 15 No 42.9% 25.0% 40.0% 10 Answered Question 21 4 25 EXTERNAL EVALUATION OF MEASURE DHS 95 Skipped Questions 2 2 4 TABLE 17: IF YOU COULD STRENGTHEN CAPACITY BUILDING ACTIVITIES FOR THE DHS IN YOUR COUNTRY, WHERE WOULD YOU RECOMMEND EFFORTS BE FOCUSED? 1. Build a strong statistics and survey research unit in the MOH by having MOH staff trained at the masters or PhD level. 2. National Institute for Statistics and Applied Economics for technical aspects, Ministry of Health and AIDS for planning, budgeting, leadership, coordination. 3. 3. Data analysis. 4. 4. Interpretation of data and use. 5. 5. The UBOS has sufficient qualified staff and recommend building their capacity for data analysis, interpretation and report writing. Also need to build capacity for secondary data analysis through hands on training. 6. 6. Data analyses and lab strengthening. 7. 7. How to improve understanding, analyzing and interpreting the survey data, how to present the data to policy makers. 8. 8. Monitoring of the quality of the activities of the DHS that the country is developing through INEI. 9. Data entry, analysis, and report writing. Also, on how to explore innovative ways to increase domestic financing for future DHS -- to institutionalize the process. 10. Technical skills in data collection, data analysis, and data processing. 11. Extensive training so that the local agency can plan and conduct DHS with very limited TA from MACRO. On-the-job training throughout the process of collecting, processing, and disseminating DHS data during the survey. 12. Forward planning, sampling, supervision/quality control, analysis, dissemination/data use. 13. Integration into formal training of either demography or Public Health studies - designing and implementing household surveys as well as analyzing DHS data sets. Whenever possible, demographers can be seconded to such institutions to build capacity of the faculties. 14. Data use at decentralized levels. 15. No particular opinion. 16. Currently, analysis of DHS is being carried out exclusively by MACRO. There is a need to build the local capacity to carry out further analysis of DHS including those involving complex indicators. The targets for such capacity building are staff at Department of Statistics and Jordanian scholars interested in relevant research. The main obstacle in this area is failure to allocate enough funding for capacity building. 17. Working with local universities and research institutions to develop curricula for capacity building on advanced data analysis and use of survey data for policy development and program improvement. 18. Focus efforts now on ability to do detailed/ complex analysis and report preparation. 19. Technical capacity of national statistical service. 20. Much more rapid and well-packaged information for states and national level. The current system is weak on both these aspects. 21. Strengthen the civil society in understanding and dissemination of data (particularly local media and local NGOs. 22. In-country DHS team should be able to carry-out sampling activities, not rely entirely on Macro; More EXTERNAL EVALUATION OF MEASURE DHS 96 individuals and organizations should be able to use DHS dataset to carry-out further analysis than doing another survey to find out the info that is feasible from analyzing DHS data. Trend analysis (based on multiple DHSs) and program specific analysis should be carried out by in-country experts. 23. Efforts should focus on the Mop and include support to local universities. EXTERNAL EVALUATION OF MEASURE DHS 97 J. PowerPoint Presentation on the Results of the Evaluation External Evaluation of the MEASURE Phase III DHS Project Richard Cornelius, Andrew Kantner, Ann Larson for MSI Overview of Evaluation Presentation  Purpose of the evaluation  Methodology  Main Findings  Recommendations 2 Purpose of the Evaluation Task 1 Provide a cost/benefits analysis of how MEASURE DHS operates, with an emphasis on the following:  How successfully the contractor is responding to varied needs of key stakeholders, and which needs are within the manageable interest of the Project. (Meeting Current Data Needs) (Stakeholder Involvement)  How successfully the contractor is providing technical assistance in survey content, design, and implementation. What technical assistance approaches (including those that may not be part of the current Project approach) might increase the contractor’s performance? (Effectiveness of Technical Assistance) 3 Purpose of the Evaluation Task 2 Provide an analysis of the feasibility/appropriateness of the capacity Building goals laid out in the contract with an emphasis on the following:  What are the priority survey design/management and data analysis/use capacity building needs of key stakeholders?  What capacity building approaches might increase the contractor’s performance in building the capacity of local institutions and individuals in survey design/management and data analysis/use? (Effectiveness of Capacity Building) 4 Purpose of the Evaluation Task 3 Provide recommendations for future directions in health data collection and survey work (based on Tasks 1 and 2). 5 Methodology Components 1. 141 semi-structured interviews with multiple stakeholder groups 2. Team visits to four Case Study countries (Bangladesh, Cambodia, Malawi, Tanzania) 3. e-Survey of USAID Field Missions and PMI Advisers 4. Document Review 6 MAIN FINDINGS 7 Task 1: Meeting Current Data Needs  Informants from every stakeholder group reported that Demographic and Health Surveys are a main source of national and regional information on demographic levels and trends and health conditions.  In the four focus countries of Bangladesh, Cambodia, Malawi, and Tanzania DHS data have been used for policy planning, health sector strategies, and monitoring program trends and outcomes. 8 9 85% 15% 0% 0% 25% 50% 75% 100% Yes, extremely well Yes, for some needs No, it is not very useful Did the DHS meet you country's need for Information (USAID Missions) 10 9.5% 33.0% 42.9% 19.0% 76.2% 90.5% 42.9% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Other STATCompiler Further Analysis Reports Dissemination Posters Summary Findings Report Final Country Report Comple Survey Data File Percentage of DHS Outputs used regularly (USAID Missions) Task 1: Meeting Current Data Needs (cont’d)  The DHS is seen as an invaluable resource by nearly all USAID informants, often referred to as the “gold standard” in health survey research. One foreign service officer called the DHS “one of the best things USAID ever got going.”  International data comparability and worldwide access to the data were cited by many AID/W, other USG, and international organization stakeholders as important distinctives of the DHS. 11 Task 1: Meeting Current Data Needs (cont’d)  Many international organizations reported they make extensive use of DHS findings, most notably UNFPA, UNICEF, UNAIDS and WHO.  The United Nations Population Division considers the DHS to be one of its most important data sources (along with population censuses and sample vital registration systems). 12 Task 1: Meeting Current Data Needs (cont’d)  Some respondents believe that too much reliance is placed on household surveys for providing core health information.  They recommend that greater investment be made in health management information systems (HMIS), especially in countries with decentralized health systems. 13  In-country informants and the e-Survey confirm that the DHS is effectively involving a wide range of relevant host country stakeholders in the identification of data needs and questionnaire design.  ICF International has demonstrated flexibility in meeting with stakeholders to accommodate requests for additional questions and special topics. 14 Task 1: Involvement of Stakeholders 15 52.40% 42.90% 4.80% 0.00% 0% 10% 20% 30% 40% 50% 60% 70% Very Responsive Responsive within Technical/Bugetary Constraints Not Responsive Don't Know Percentage who say DHS responsive to stakeholder needs (USAID Mission) Task 1: Involvement of Stakeholders (cont’d)  There has been a successful effort in recent years to seek other donor funding for DHS surveys.  But the tradeoffs include some loss of USAID technical control of the surveys as more costs are covered by other donors.  Many informants in AID/W and at ICF International also expressed serious concerns that growing data demands are resulting in survey interviews that are too long, and sample sizes that are too large, creating very real risks of deterioration of data quality. 16 Task 1: Involvement of Stakeholders (cont’d)  According to ICF International and USAID/W informants, the DHS Project generally has tried to involve local implementing agencies (LIAs) in every stage of survey operations.  However, in two Case Study countries (Malawi and Bangladesh) LIAs felt they were not properly involved in secondary data editing, tabulation, and data dissemination activities. 17 Task 1: Effectiveness of Technical Assistance  In the e-Survey, 70% of USAID Missions said their country received the full package of 10-12 technical assistance visits by ICFI experts, in addition to some HQ technical support by email.  Based on our case study interviews and e-Survey findings from USAID missions, the short term in-country TA received high marks overall.  However, the number and types of ICF International TA visits are often determined primarily by ICF International without a full assessment or discussion with the implementing agency regarding their capabilities and technical assistance requirements; and  In the e-Survey, 54% of Missions said they would like to adjust the level or types of TA provided for the next survey. 18 Task 1: Effectiveness of Technical Assistance (cont’d)  In addition to TA visits by ICF International staff, other models of TA have included use of local or regional resident advisors and consultants with prior DHS experience.  In case study countries, stakeholders were very positive about these TA inputs, saying that they offered more opportunities for mentoring and learning.  To date, these models have been used in addition to ICF International TA visits rather than substituting for TA visits, so there has not yet been any cost savings. 19 Task 2: Effectiveness of Capacity Building In the MEASURE Phase III DHS RFP/Contract:  Capacity building (CB) is one of six main Results to be achieved under the contract; and  Capacity building is meant to address all stages of the survey operations continuum.  Country-level CB activities are meant to be Mission-funded. 20 Task 2: Effectiveness of Capacity Building (cont’d) In answer to the Task 2 main question: The evaluation team finds that the capacity building goals in the contract are feasible, but are not yet being fully implemented at the country level. 21 Task 2: Effectiveness of Capacity Building (cont’d)  Over the years, the DHS has had a positive impact on survey capacity building in many countries. – Egypt, Indonesia, the Philippines and Thailand are successfully managing their own surveys without significant TA from ICF International. – Colombia is also implementing its own surveys, and is in fact providing south-to-south TA – Countries such as Bangladesh, Kenya, Nepal, Peru, Tanzania, and Uganda have made significant gains in survey capacity.  DHS experience in LAC (Peru and Ecuador) shows that graduating countries from DHS assistance needs to be a gradual planned process. 22 23 9.5% 0.0% 0.0% 23.8% 38.1% 28.6% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% Don't Know Deterioriation No Growth Yes, Limited Growth Yes, Some Growth Yes, Large Growth Have you seen growth in capacity to conduct the DHS (USAID Mission) Task 2: Effectiveness of Capacity Building (cont’d) e-Survey results point to mixed results regarding CB implementation:  Only 57% of countries had a line item for capacity building in their most recent survey budget  62% of respondents were very satisfied or satisfied with ICFI’s capacity building activities.  When asked to name the most important thing needed to further strengthen DHS CB activities in their country, not one Mission mentioned the need for more funding. 24 Task 2: Effectiveness of Capacity Building (cont’d)  In the Case Study countries, almost every host government stakeholder indicated a need for more capacity building, especially related to sampling, data processing, data utilization, and further analysis.  Instead of seeing capacity building as a trade-off with quality, they saw strengthened capacity for survey implementation and analysis as primary indicator of quality.  To be most effective, in-country planning for capacity building needs to occur very early in the survey design process, and be included in the survey plan and budget. 25 Selected Recommendations 26  In each country, ICF International should develop a country plan for data dissemination, utilization and further analysis during the survey design stage of the survey, in collaboration with the USAID Mission and the National Steering Committee.  ICF International should encourage the development of less technical thematic policy briefs that would distill and discuss the policy/program implications of DHS findings. In most cases, these policy briefs should be written by local experts, and printed and widely disseminated in-country. 27 Task 1: Meeting Current Data Needs Task 1: Involvement of Stakeholders  To the extent possible, the implementation of every stage of a DHS survey (including sampling, data processing and data analysis and report writing) should be done in￾country, with the fullest possible participation of host country staff. 28 Task 1: Effectiveness of Technical Assistance  Regardless of which models of TA are employed, TA provided by ICF International should always include full engagement of host country counterparts to promote mentoring and learning. Every TA visit is a capacity building opportunity.  During TA visits, ICF International staff and consultants should never do the work for the implementing agency; they should always do the work with the implementing agency. 29 Task 2: Effectiveness of Capacity Building  Plans for supporting further dissemination, utilization, and analysis of DHS data in each country should be implemented in ways that will also contribute to building host country capacity.  Planning for capacity building activities should occur during the survey design phase of surveys, and in discussion with the USAID Mission and the National Steering Committee, so the costs for capacity-building activities can be included in the survey budget. 30 Task 2: Effectiveness of Capacity Building (cont’d)  ICF International should consider more widespread use of resident advisors and part-time regional consultants with the goal of relieving some of the TA burden on ICF International HQ staff.  ICF International should consider the possibility of opening at least one regional office (e.g., in sub Saharan Africa). These could be staffed with full-time advisors, including the best and brightest among the regional consultants ICF International has already been using, as well as some ICF International HQ advisors. 31 Task 2: Effectiveness of Capacity Building (cont’d)  A plan for graduation of a country from the DHS assistance should be developed jointly by ICFI, the USAID Mission, and host government stakeholders up to five years in advance.  Graduation plans should allow for a smooth transition to post- DHS support so that necessary funding can be secured to support the sustainability of household-based health surveys. 32 Task 3: Future Directions for Demographic and Health Data Collection  USAID/GH should diversify its available mechanisms for conducting population-based surveys. Specifically, the AIS, MIS, and SPA surveys (and related modules) should taken out of the DHS, and each re-implemented as separate survey programs.  This would result in 4 more-specialized survey programs, each with full control of developing/managing its own portfolio of surveys and/or other special studies, its own questionnaire content, and the frequency and timing of its own surveys. And the growing data quality risks in DHS, caused by the overloading of subject matter content onto that project and its questionnaires, would go away.…a win-win outcome for all. 33 Future Directions for Demographic and Health Data Collection (cont’d)  USAID should consider expanding its support for improvement of Health Management Information Systems. Improvement of HMIS data could reduce the demand for district-level disaggregation of population-based survey estimates and lessen the demand for larger DHS samples.  These efforts would lead to an improvement of the locally- collected data upon which health program management decisions are made. 34 Future Directions for Demographic and Health Data Collection  ICF International should do fund-raising and resource mapping for surveys well in advance, so there are no delays in implementation.  There should also be a post-survey lessons learned discussion on survey operations that could inform planning for the next survey. 35 Thank you very much. 36