PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU) DATA QUALITY REVIEW AND MIDTERM ASSESSMENT JULY 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Jim Heinzen, Archer Heinzen, Luz Illescas, and Susana Guevara through CAMRIS International RFTOP No. 527–10-005, Task Order under the GHS-I-00—07-00015/AID-527-TO-10-00001, TASC III, Data Quality Review and Assessment for the Peru MCC Threshold Program. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU) DATA QUALITY REVIEW AND MIDTERM ASSESSMENT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 10-01-1018) is available in printed or online versions. Documents are available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from: CAMRIS International 6931 Arlington Road, Suite 575 Bethesda, Maryland 20814 Tel: (301) 770-6000 Fax: (301) 770-6030 www.camris.com This document was produced for review by the United States Agency for International Development. It was prepared by CAMRIS International Order under the GHS-I-00—07- 00015/AID-527-TO-10-00001, TASC III. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT i CONTENTS ACRONYMS .......................................................................................................................................................iii I. THRESHOLD PROGRAM BACKGROUND ................................................................................1 II. DATA QUALITY REVIEW, JUNE–JULY 2010 .............................................................................3 Executive Summary ..............................................................................................................................3 Methodology ..........................................................................................................................................5 Limitations of the Study ......................................................................................................................7 Data Quality Review ............................................................................................................................8 III. COMPONENT 1: ANTICORRUPTION ......................................................................................9 The Baseline Study ...............................................................................................................................9 Analysis of Indicators ........................................................................................................................ 10 General Recommendations on Anticorruption Indicators ..................................................... 21 IV. COMPONENT 2: IMMUNIZATIONS ....................................................................................... 23 Demographic Analysis of the Family Health Survey and Baseline Study .............................. 23 Analysis of Indicators ........................................................................................................................ 24 Observations and Conclusions ...................................................................................................... 29 V. MID-TERM ASSESSMENT OF THE THRESHOLD PROGRAM, JUNE–JULY 2010 ......... 31 Executive Summary ........................................................................................................................... 31 Purpose of the Assessment ............................................................................................................ 32 Methodology ....................................................................................................................................... 33 Component 1. Anticorruption ..................................................................................................... 33 Component 2. Immunizations ....................................................................................................... 46 TABLES TABLE II.1. SUMMARY OF INDICATORS .................................................................................................3 TABLE II.2. VISITS DURING THE ASSESSMENT ......................................................................................6 TABLE III.1. GENERAL DATA QUALITY: FINDINGS AND RECOMMENDATIONS: ............. 10 TABLE III.2. INDICATOR 1.1: FINDINGS AND RECOMMENDATIONS ..................................... 11 TABLE III.3. INDICATORS 1.2–1.4: FINDINGS AND RECOMMENDATIONS RELATED TO CORRUPTION REPORT PROCESSING ........................................................................................... 12 TABLE III.4. INDICATOR 1.5: FINDINGS AND RECOMMENDATIONS ..................................... 14 TABLE III.5. INDICATORS 1.6–1.7: FINDINGS AND RECOMMENDATIONS ........................... 15 TABLE III.6. INDICATOR 1.8: FINDINGS AND RECOMMENDATIONS ..................................... 16 TABLE III.7. INDICATOR 1.9: FINDINGS AND RECOMMENDATIONS ..................................... 17 ii PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE III.8. INDICATORS 1.10–1.11: FINDINGS AND RECOMMENDATIONS ....................... 18 TABLE III.9. INDICATOR 1.14: FINDINGS AND RECOMMENDATIONS .................................. 19 TABLE III.10. INDICATOR 1.15: FINDINGS AND RECOMMENDATIONS ................................. 21 TABLE IV.1. INDICATOR 2.1: FINDINGS AND RECOMMENDATIONS ..................................... 24 TABLE IV.2. INDICATOR 2.2: FINDINGS AND RECOMMENDATIONS ..................................... 25 TABLE IV.3. INDICATOR 2.3: FINDINGS AND RECOMMENDATIONS ..................................... 27 TABLE IV.4. INDICATOR 2.4: FINDINGS AND RECOMMENDATIONS ..................................... 28 APPENDIXES The appendixes are published on the following pages in Volume II. APPENDIX 1. RESULTS REPORTING TABLE (RRT) .............................................................................. 1 APPENDIX 2A. ASSESSMENT OF INDICATORS AND MILESTONES (ENGLISH VERSION) ..................................................................................................................................... 11 APPENDIX 2B. ASSESSMENT OF INDICATORS AND MILESTONES (SPANISH VERSION) ...................................................................................................................................... 87 APPENDIX 3. OUTLINE OF STRUCTURED INTERVIEWS WITH COUNTERPARTS AND IMPLEMENTERS FOR ANTI-CORRUPTION COMPONENT (SPANISH VERSION) ..... 177 APPENDIX 4. OUTLINE OF STRUCTURED INTERVIEWS FOR THE IMMUNIZATION COMPONENT (SPANISH VERSION) ..................................................................................................... 179 APPENDIX 5. OUTLINE FOR FOCUS GROUPS (SPANISH VERSION) ...................................... 181 APPENDIX 6. CALENDAR OF ACTIVITIES OF THE CONSULTANCY (ENGLISH VERSION) ................................................................................................................................... 183 APPENDIX 7. LIST OF PEOPLE INTERVIEWED (SPANISH VERSION) ....................................... 185 APPENDIX 8. DATA FLOW DIAGRAMS (SPANISH VERSION).................................................... 193 APPENDIX 9. BIBLIOGRAPHY (SPANISH VERSION) ....................................................................... 197 APPENDIX 10. SCOPE OF WORK (ENGLISH VERSION) ............................................................... 203 APPENDIX 11. CONECTA DOCUMENT (SPANISH VERSION) .................................................. 219 APPENDIX 12. CONECTA SURVEY INSTRUMENT (SPANISH VERSION) .............................. 229 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT iii ACRONYMS The evaluation team and USAID/Peru agreed to use the acronyms used in the Peru Threshold Program Results Reporting Table in this report. In some cases, as is the practice of USAID/Peru, the Spanish acronym is used for both English and Spanish. In cases where the PTPRRT does not list an acronym, the English equivalent has been derived from the Peruvian source for the Spanish. AC Anti-corruption BFR Budgeting for results CNM National Judicial Council (Consejo Nacional de la Magistratura) DGSP General Directorate of People’s Health DHS Demographic and Health Survey DIRESA Regional Health Directorate (Dirección Regional de Salud) DPT3 Diphtheria, pertussis, and tetanus vaccine, third dose DQR Data quality review ESNI National Health Immunization Strategy (Estrategia Sanitaria Nacional de Inmunizaciones) GOP Government of Peru GOSI General Office of Statistics and Informatics HIS Health information system HU/HC High use/high corruption IB Itinerant brigades ICITAP U.S. Department of Justice International Criminal Investigative Training Assistance Program ICO Internal Control Office, Office of the Comptroller General IDB Inter-American Development Bank IIIS Integrated Immunization Information System INEI National Statistical Institute (Instituto Nacional de Estadística e Informática) IT Information technology MCC Millennium Challenge Corporation MEF Ministry of Economy and Finance MININTER Ministry of Internal Affairs (Ministerio del Interior) MOH Ministry of Health MMR Measles, mumps, and rubella vaccine MSI Management Systems International NGO Nongovernmental organization OCG Office of the Comptroller General of the Republic iv PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT OCMA Office of Internal Control of the Judiciary (Oficina de Control de la Magistratura) ODECMAs Decentralized Offices of Internal Control of the Judiciary ODSIS Decentralized Offices for Comprehensive Health Insurance (Oficinas Descentralizadas del Seguro Integral de Salud) OGEI General Office of Statistics and Data Processing (Oficina General de Estadística e Informática) PAHO Pan American Health Organization PCM Presidency of the Council of Ministers PD Public defender PDA Personal digital assistant RENIEC National Registration of Identification and Marital Status (Registro Nacional de Identificación y Estado Civil) RRT Results Reporting Table SIGA Management and Administrative Information System (Sistema de Informacion Gerencial y Administrativa) SIS Comprehensive Health Insurance (Seguro Integral de Salud) SOAG Strategic Objective Agreement SOW Scope of work TA Technical assistance USAID United States Agency for International Development PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 1 I. THRESHOLD PROGRAM BACKGROUND In November 2007, the Millennium Challenge Corporation (MCC) approved the Government of Peru (GOP) proposal for the Peru MCC Threshold Program. In June 2008, the U.S. Agency for International Development (USAID) signed an agreement to provide $35.6 million in assistance for a threshold program with two components: (1) reducing corruption in public administration, and (2) increasing immunization rates. Component 1 of the Threshold Program has five objectives for reducing corruption: A. Reduce the prevalence of bribes paid by citizens and lawyers to the judiciary. B. Reduce the prevalence of bribes paid by citizens to the police. C. Reduce the prevalence of bribes paid by citizens and businesses to public servants. D. Increase public awareness about corruption and the government’s anticorruption efforts. E. Increase the capacity of citizens to use public information to effectively assess and monitor government transparency and accountability. Three types of interventions support these objectives: fostering awareness, enhancing enforcement, and reducing opportunities for corruption. Component 2 of the Threshold Program has three objectives for increasing immunization coverage: A. Increase basic immunization (diphtheria, pertussis, and tetanus vaccine, third dose [DPT3] and measles, mumps, and rubella vaccine [MMR]) of children in rural areas using mobile teams to provide comprehensive health services to excluded and dispersed populations. B. Build up national logistics systems for vaccine management and provide training in how the systems operate. C. Strengthen the Integrated Immunization Information System (IIIS) in both centralized and decentralized MOH systems to improve the accuracy and speed the flow of information for monitoring, management, and follow-up of plans for routine and supplementary immunizations to improve immunization coverage. Interventions in support of these objectives include: providing equipment and training for itinerant brigades (IBs) that bring health services to remote areas; improving information systems; and strengthening the Ministry of Health (MOH) and regional immunization supply chain systems. The Threshold Program is implemented by seven1 Peruvian government counterparts and four implementers. The GOP counterparts are: Office of Internal Control of the Judiciary (OCMA) Ministry of the Internal Affairs (MININTER) Office of the Comptroller General of the Republic (OCG) 1 In June 2010 there were only the first five GOP counterpart institutions; the last two (CNM and the Office of Internal Control of the Attorney General) joined a month later. 2 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Ombudsman’s Office Ministry of Health (MOH) and regional governments National Judicial Council (CNM) Office of Internal Control of the Attorney General The institutions implementing the program are: Management Systems International (MSI) U.S. Department of Justice International Criminal Investigative Training Assistance Program (ICITAP) ProEtica Abt Associates (Abt) In June 2010, USAID signed a contract with CAMRIS International to carry out a data quality review (DQR) and assess program achievements to date, focusing on both sustainability and opportunities for cooperation with and integration of Threshold Program activities into other programs. CAMRIS International hired Jim Heinzen, MBA, as team leader, with Archer Heinzen, PhD, Susana Guevara, MA, and Luz Illescas, MD, as members of the team and Gabriela Torres providing administrative support. The evaluation team began activities in Peru on June 7. This report sets out an analysis of data quality followed by an assessment of the Threshold Program. Each section first addresses Component 1, Anticorruption, then Component 2, Immunizations. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 3 II. DATA QUALITY REVIEW, JUNE–JULY 2010 EXECUTIVE SUMMARY The data quality review (DQR) report covers project indicators and milestones and sets forth the findings. The Results Reporting Table (RRT) lists the program´s 19 MCC indicators and describes targeted end results, baselines, and the schedule of milestones for measuring program results. The RRT in Appendix 1 is from USAID´s quarterly report to the MCC for the second quarter of FY 2010, which covers just over half of the projected life of the program. Activity milestones are almost exclusively the responsibility of program counterparts. Although GOP counterparts often require complementary activities and actions, the RRT does not report on these. The DQR assesses the quality of the indicator data in terms of its validity, reliability, timeliness, precision, and integrity. Table II.1lists the 19 MCC indicators and briefly describes their status. TABLE II.1. SUMMARY OF INDICATORS MCC Indicator Data Quality 1 Anticorruption Component 1.1 Percentage of judiciary system users that know how to use reporting and control of corruption mechanisms. Reliability limitations due to sampling method (nonprobabilistic quota sampling and random selection). The survey company, GfK Conecta, contests this observation (see Appendix 11). The sampling observation also applies to indicators 1.5, 1.8, 1.9 and 1.14. 1.2 Number of days required by OCMA and ODECMAs [Decentralized OCMA Offices] to process a corruption report (for judges and judiciary employees). (Strategic Objective Agreement [SOAG] indicator). Timeliness and integrity limitations. Data are dependent on a new information system the Threshold Program is installing. The system design meets data quality standards for the indicator. 1.3 Percentage of sanctions recommended by ODECMAs confirmed by OCMA after appeal. Timeliness limitations. Data are dependent on a new information system the program is installing. The system design meets data quality standards for the indicator. 1.4 Percentage of OCMA decisions where a sanction is recommended that are executed by the National Judicial Council (CNM) (SOAG) Reliability limitations. Data requirements are well defined, but data sources give contradictory information. Because the CNM is being included in the program, it is expected that issues of data reliability will be resolved and the indicator will be validated. 1.5 Percentage of citizens who are willing to make a complaint. Reliability limitations due to sampling method used. Survey question used can be leading. 1.6 Number of days between the beginning of the disciplinary procedure and its final resolution (SOAG) Timeliness limitations. The indicator is well defined but beyond establishing a baseline, data collection is on hold pending approval of new regulations 4 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE II.1. SUMMARY OF INDICATORS MCC Indicator Data Quality 1.7 Percentage of disciplinary reports that are processed within the timeframe established by the law. Timeliness limitations. Indicator is well defined except that it should state a specific number of days, not general time frames established by the law. No data have been generated, Implementation is behind schedule, and ICITAP may request a change in the indicator. 1.8 Percentage of citizens that know the improved traffic procedures. Reliability limitations due to sampling method used. 1.9 Percentage of citizens who are aware of the Comptroller General’s role in anticorruption activities. Reliability limitations due to sampling method used. Reliability may be compromised because the survey question can be leading. 1.1 0 Time required by OCG to investigate an allegation of corruption (SOAG). Validity issues. Indicator not clearly defined. Specifying a specific number of days rather than a percentage reduction would validate the indicator. 1.1 1 Number of Internal Control Offices (ICOs) processing corruption reports according to revised OCG Guidebook. No issues identified. The original number of ICOs has been reduced due to reprogramming. 1.1 2 Percentage of OCG’s implementable recommendations on selected internal control processes fully implemented. Eliminated at the request of the Comptroller due to change of activities. 1.1 3 Percentage of processes involved in the provision of high use and high corruption (HU/HC) services streamlined. Eliminated at the request of the Comptroller due to change of activities. 1.1 4 Percentage of citizens that recognize the Ombudsman as an institution that monitors GOP anti-corruption (AC) initiatives. Reliability may be compromised because the survey question used can be leading. Reliability limitations due to sampling method used. 1.1 5 Number of actors actively participating in the anticorruption network. No quality issues identified. Reporting is on schedule. 2 Immunizations Component 2.1 Coverage of measles (MMR) and third dose of diphtheria, pertussis and tetanus vaccine (DPT3) in eight priority regions. No quality issues identified. Reporting is on schedule. 2.2 Percentage of health facilities with adequate vaccine supply. The last quarterly report corrected a minor measurement error. No other quality issues identified. 2.3 Time for health centers to receive information from itinerant brigades (IBs). No quality issues identified. Reporting is on schedule. 2.4 Registry errors in data reporting to Regional Health Directorates (Direcciones Regional de Salud or DIRESAs). No quality issues identified. Reporting is on schedule. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 5 In Anticorruption, an early finding resulted in the counterpart dropping indicator 1.10 and pausing activities related to indicators 1.11, 1.12, and 1.13. Changes to indicator 1.7 have been proposed but not yet accepted. Other actions on indicators approved earlier in the project have resulted in continued discussion of some indicators or their measurement. Anticorruption data, aside from the survey data, suffers from the complexity of having two implementers (MSI and ICITAP) and six counterparts (OCMA, MININTER, OCG, Office of Ombudsman, CNM, and the ICOs of the Attorney General). While in theory, the quality of an indicator is independent of the number of parties, from a practical standpoint a larger number of people dealing with the indicator and milestone definitions impacts negatively on its application. Moreover, the more people, the more turnover, so that new people must constantly be oriented. For Component 1 there has been turnover of key people in both MSI and counterparts. ProEtica, a civil society anticorruption implementer, has a well-developed information system. Immunizations data were generally of higher quality than the anticorruption data, as can be seen from the comments for the last four indicators on Table II.1. This may partly be attributable to the facts that the health system has better-established data systems and the component has a single implementer (Abt) and a single counterpart (MOH). Data quality deficiencies identified during the program design stage were addressed by a significant investment in new information systems. Abt is installing these systems but they are not yet operational. As a result, the data needed to track some indicators are based on an information system that has yet to be installed. The immunizations component uses four indicators that meet standards for validity, reliability, timeliness, precision, and integrity. The completion of milestones as reported in the third quarter FY 2010 RRT was verified. However, some milestones have been changed in the course of implementation (number of IBs and regions having the IIIS) suggesting that the milestones are being used to describe processes rather than targets. Each milestone was defined at the beginning of the program as an intermediate result or as a commitment of the counterpart (MOH in the case of immunizations) to make progress as planned. Milestones partially met are documented in the monitoring section of the quarterly report. The MCC RRT provides for the reporting of delayed results in subsequent reports, accompanied by an explanatory note. METHODOLOGY The evaluation team submitted its work plan to USAID on June 14. At the meeting USAID asked for clarification on several points and added others, which were submitted electronically; one of the consultants followed up with USAID by telephone. To organize the work efficiently, the team leader assigned each team member primary responsibility over a group of indicators. Susana Guevara focused on five indicators (1.1, 1.5, 1.8, 1.9 and 1.14), the data source for which is the baseline of the national survey and a special survey. Team leader Jim Heinzen focused on evaluation of indicators for the judiciary (1.2, 1.3, and 1.4) and the Comptroller (1.10, 1.11, 1.12, 1.13). Archer Heinzen led evaluation of the indicators for the MININTER and ProEtica (1.6, 1.7, and 1.15), and Luz Illescas led the evaluation of the immunization indicators (2.1, 2.2, 2.3, and 2.4). Although each team member was responsible for evaluation of specific indicators, all reviewed all indicators. Subteams of two consultants each, one Peruvian and one American, attended all meetings with counterpart and implementing organizations. From June 23 to June 25, two team members traveled to Ica. From June 29 to July 2, team members traveled in pairs to Huamanga (Ayacucho) and Chachapoyas (Amazonas). Throughout June 9 to July 5, the team held meetings in Lima, 6 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Analysis of Baseline Studies For Component 1, Anticorruption, as a baseline the evaluation team reviewed documents provided by USAID, MSI, and its CONECTA partner, including the original questionnaire and baseline results. The team supplemented the document review by meeting with CONECTA Associates to clarify issues not covered in the documents. The meeting reviewed questions and sampling methodology, margin of error, interviewer training, data processing, and reporting. For Component 2, Immunizations, the team reviewed both Abt documentation and the methodological design of the Diagnostic Study of the MOH National Immunization Strategy. The team also met with Abt’s head of monitoring and evaluation to clarify the study methodology and better understand its design. Analysis of the Process and Other Indicators To support the DQR, Mariela Zapata of the Office of the Presidency of the Council of Ministers (PCM), sent letters of presentation to the officials in charge of the Threshold Program government counterparts. The team leader then sent letters to each organization explaining the task and requesting assistance. At meetings on June 15 and 16, the team met with MSI staff for guidance on the government counterparts and the data they are (or are not) collecting. The team met on June 9, 17, and 18 with Abt staff and thereafter maintained constant e-mail communication to obtain data and guidance about development of Component 2, Immunizations. Among the materials the evaluation team drafted for use in data collection were a format for assessing indicators based on the format used in the first data quality assessment of indicators conducted by MSI; a format for evaluating milestones; an interview protocol: and a guide for focus groups. Evaluation forms for indicators and milestones (Appendices 3 and 4) were adapted during interviews as experience dictated. As an initial step in the DQR, implementers and their government counterparts were asked for data flow diagrams. Some provided them, others did not. Table II.2 summarizes visits made during the evaluation. TABLE II.2. VISITS DURING THE EVALUATION Counterpart/Indicator Number Interviews, Data Review and Focus Group Lima Field: Ica, Ayacucho, and Amazonas Component 1– Anticorruption Judiciary/OCMA OCMA ODECMA and work sessions Ministry of Internal Affairs (MININTER) MININTER: Interview and data review Although ICITAP conducted training only in Ayacucho, we interviewed representatives of the police during all three￾site visits to learn about their anticorruption activities. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 7 TABLE II.2. VISITS DURING THE EVALUATION Counterpart/Indicator Number Interviews, Data Review and Focus Group Office of the Controller General (OCG) OCG Comptroller of the Republic and regional government ICOs Ombudsman Office of the Program for Public Ethics, Prevention of Corruption, and Public Policy Offices of the Public Ombudsman Civil society ProEtica Lima: Interview and data review on participation Networks: Discussions / focal groups with network members Component 2 – Immunizations Ministry of Health (MOH) Abt: Interview and review of data on vaccinated children, provision of vaccines, timeliness of data entry by IBs, and reporting of errors in records. National Health Immunization Strategy (ESNI): PAHO (Pan American Health Organization) interview Abt Offices in Ica, Ayacucho, and Amazonas. Regional Health Offices of Ayacucho, Ica, and Amazonas. Interviews and visits to health facilities and data review. IB team interview. Regional government interview. Oficinas Descentralizadas del seguro Integral de Salud (Decentralized Office for Comprehensive Health Insurance) ODSIS interview LIMITATIONS OF THE STUDY Factors influencing the conduct of the study include Difficulty in arranging meetings, despite notification of implementers and counterparts. The evaluation team assigned one member almost exclusive responsibility for scheduling appointments who worked closely with Mariela Zapata of the PCM. Three of the implementers provided considerable collaboration but one was unavailable for three weeks. A key subcontractor of one implementer was unavailable to meet for several weeks and was still delivering material three days before the DQR was presented. Despite repeated phone calls, two counterparts did not provide appointments until two days before the deadline for submission of the study. Some of those interviewed failed to forward documents promised during meetings. Several holidays made it difficult to schedule field visits. 8 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT DATA QUALITY REVIEW The DQR focuses on the indicators, the quality of data used to measure them, and the milestones representing activities that caused positive change in the results expected for the indicators. What follows is an analysis of indicators for both components and a review of the data and milestones for each. Detailed assessment of each indicator and milestone can be found in Appendix 2. Recommendations are focused on ways to mitigate deficiencies found in the indicators and milestones by adding new indicators and milestones, modifying current ones, eliminating some, or modifying definitions. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 9 III. COMPONENT 1: ANTICORRUPTION THE BASELINE STUDY This study of the population 18 years and older established the baseline for the program’s perception indicators (1.5, 1.8, 1.9, and 1.14). Now underway is a similar midterm study to measure changes in perception because of program interventions. USAID is planning a final study as the program moves to a close. The baseline study carried out by the local firm CONECTA, was a household survey. The 136- item questionnaire was reviewed and tested several times. The survey universe was the population 18 years and older, taken from the 2007 census conducted by National Statistical Institute (INEI) (total: 17,377,861 people). The survey sample size was 8,336 persons. The sample used a combination of sampling techniques: convenience, quota, and random selection. It is a nonprobabilistic sample because it assigns sex and age quotas for the people to be interviewed. Because this technique does not comply with probability sampling methods, it is not possible to calculate sampling error for the values found or to apply techniques of inferential statistics. Consequently, the results of this study represent only the views of a specific sample of more than 8,000 people and do not permit national inferences, a limitation of the design. A sample must be random to make valid inferences about the universe; assigning quotas for men and women and by age group results in a stratified sample, which can result in biased selection. It would be possible to correct the sample by weighting it in terms of the proportion of the sample to the census population in terms to sex and age. However, because age was recorded only in terms of large age ranges at the time of the interview, it is not possible to make this correction, especially if there was a bias toward a specific age group when respondents were selected. The sample was stratified by size of population clusters, and then provinces were selected in proportion to their size. Each site was assigned a quota of interviewees, in both quantity and characteristics (age and sex). Households were selected using the random sampling frame of INEI for Lima and the Peruvian Association for Market Research for other cities. Also, houses were selected using a systematic jump (every third house), although, the evaluation team considers that this makes no sense when using a quota sample. The team conducting the fieldwork consisted of 387 interviewers, 37 supervisors, and 35 team leaders. CONECTA reported that the team received two days of training: one on general technical aspects of surveys and the other on practical exercises for implementing the surveys for this study. Given the size of the questionnaire (136 questions), the evaluation team considers the training time to have been too short. Training information was uncorroborated; while the evaluators obtained the list of interviewers, contact information for them was lacking. Without a field manual to see how the field staff operated or the ability to interview field personnel, it was not possible to randomly confirm the quality of the survey work. CONECTA reviewed the first draft of this report and took issue with some of the findings. The CONECTA document (see Appendix 11) argues that several companies use its multistage sampling. It describes studies by LAPOP, Latinobarometro, and Apoyo. However, the evaluation team comments relate to the type of sampling used for this program, which is nonprobabilistic, not to multistage sampling. CONECTA believes that the methodology applied does allow measurement that produces the same results as if the sample had been totally probabilistic. 10 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT It took more than two weeks for the evaluation team to arrange a meeting with CONECTA, and its submission of requested information was slow. The last arrived three working days before the DQR was presented. Other issues raised in the CONECTA document are addressed in relevant sections of the data review. TABLE III.1. GENERAL DATA QUALITY: FINDINGS AND RECOMMENDATIONS: Findings Recommendations – The sample design uses a nonprobability sampling technique based on quota sampling and random selection. This means that the results cannot be validly projected to the national level. The results of the survey should be taken as the responses of a specific sample. – MSI should document the research: methodological design, applied surveys, databases, questionnaires, etc. – MSI must accompany the company doing the study to ensure the quality of the information gathered, data processing, and reports. ANALYSIS OF INDICATORS Indicators are presented in the same order as in the RRT Government counterpart: Office of Internal Control of the Judiciary (OCMA) and decentralized Offices of Internal Control of the Judiciary (ODECMAs) Implementer: MSI Actions responding to Component 1, Objective A: Reduce the prevalence of bribes by citizens and lawyers to the judiciary. Indicator 1.1: Percentage of judiciary systems users that know how to use reporting and control of corruption mechanisms. A national baseline study of lawyers and litigants is the data source for this indicator. The study asked 30 questions. The universe was the total number of files submitted to the judiciary from April to December 2008 (211,494 files). The sample was probabilistic, with a confidence level of 95% and a margin of error of 2.75%. The sample size was 1,266 persons. The sample was selected using a multi-step process: first cities were selected in proportion to size, then the cities with the highest legal caseloads were selected, and from among these, the sample was calculated taking into account proportional distribution. This ladder type of selection removes sample randomness. Then, to select sampling units (respondents) a systematic jump of every third case was applied. The premise was that the interviewers were constantly touring the judiciary building to select respondents. The evaluation team could not corroborate control of the random process for sampling units. A team of 26 interviewers conducted the survey. The evaluators were unable to verify the quality of training they received because the evaluators had access only to the list of interviewers with no contact information. CONECTA professionals mentioned that the field staff was trained over two days: one day on general aspects of surveys (training given to all new personnel) and a half-day of training on the implementation of specific surveys. The evaluators feel that the amount of time for training for a survey of this kind should have been greater because of the number of questions (30) and the type of sampling used. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 11 Documentation was not available to affirm that field supervisors made random checks on the surveys. The field manual does not specify this as a function of the field assistant or provincial coordinator. It was also not possible to cross-check information between the database and the surveys conducted, which would have made it possible to detect errors.2 TABLE III.2. INDICATOR 1.1: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – The indicator does not measure what it is expected to measure (knowledge of how to use control-of-corruption mechanisms). Three questions were used in constructing the indicator: P5. Did you know or not know that the judicial branch has an office called OCMA (Office of Internal Control of the Judiciary) devoted specifically to receiving complaints and to control and punish acts of corruption that occur in the courthouse and courtrooms? – P13. Did you know you can file a grievance or complaint by filing a written complaint at the reception desk of the OCMA or ODECMA? – P21. Did you know that the functions of the OCMA include investigating breaches of discipline by judges and judicial employees? – Two of the questions probe for knowledge of the functions of OCMA and ODECMA and the second question asks about knowledge of the written complaint mechanism. There is no direct inquiry about knowledge of how to use the complaint mechanisms. – At this time, because it is too late to change either the indicator or the survey question, there is no recommendation. Reliability – The sample was probabilistic; the sites selected were larger cities with the greatest legal caseloads, which compromises randomization. – Respondents (lawyers and litigants) were selected using fraction sampling, – Ensure that locations and sample units are chosen randomly. Document the: methodological design, surveys conducted, databases, questionnaires, etc. – Accompany the firm doing the study to ensure quality in information collection, data 2 In the document in which CONECTA comments on the evaluation—not on the implementer— it is noted that surveys are not kept for more than one year. The contractor, in this case MSI, should retain all sources of verification until the intervention is finished. 12 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE III.2. INDICATOR 1.1: FINDINGS AND RECOMMENDATIONS Findings Recommendations the application logic for which should be supported. The evaluation team could not corroborate control of the random process by sampling units. processing, and reporting. – Check the consistency of data in the field and entry of data into the database to ensure data quality, which would highlight lack of data and inconsistent data. Milestones – Training: This is verifiable. It offers workshop and training material design, and workshop programs are on the OCMA Web site. – Review the materials to ensure that they are easily understood by the public to whom they are addressed (e.g., leaders of grassroots organizations) – People attending the workshops. This is verifiable. There are Excel lists of participants that are on the OCMA Web site. – Build a database of participants Indicator 1.2: Number of days required by OCMA and ODECMAs to process a corruption report (for judges and judiciary employees.) Indicator 1.3: Percentage of sanctions recommended by ODECMAs that are confirmed by OCMA after appeal. Indicator 1.4: Percentage of OCMA decisions where a sanction is recommended that are executed by the National Judicial Council (CNM). (SOAG) Information for the second group of indicators comes from the records of the internal information system of OCMA and the ODECMAs, which will be replaced by a new information system developed by the Threshold Program. This system will encompass the 29 ODECMAs established nationally. Estimated data for the baseline for Indicators 1.2 and 1.3 come from the current data system. The baseline for indicator 1.4 is calculated using statistics published on the OCMA and CNM Web sites. The new information system is now being tested. TABLE III.3. INDICATORS 1.2–1.4: FINDINGS AND RECOMMENDATIONS RELATED TO CORRUPTION REPORT PROCESSING Findings Recommendations – The principal action related to Indicators 1.2, 1.3, and 1.4 is installation of a modern information system for the 29 ODECMAs and OCMA. MCC guidelines require that RRT include process indicators to track implementation. The process is defined by the milestones documenting implementation status. No process milestones were found for Indicators 1.3 and 1.4. – Add to Indicator 1.2 the objective of the outcome of the phrase ―ODECMA with the system installed.‖ – Modify the milestones of Indicator 2 to better relate to the information system being installed. ―Milestone = ODECMA with the system = 150 days FY2010, fourth quarter.‖ PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 13 TABLE III.3. INDICATORS 1.2–1.4: FINDINGS AND RECOMMENDATIONS RELATED TO CORRUPTION REPORT PROCESSING Findings Recommendations – Indicator 1.2 measures the performance of the information system being implemented. There is some cause and effect, though distant, between the intervention and the indicator. – Indicator 1.3 is also a performance indicator, but it is difficult to identify a cause-effect relationship between OCMA milestones and the indicator. The logic is that the new system will accelerate processing and mitigate opportunities for corruption, leading to a higher confirmation rate. Although the indicator is interesting, because many factors could explain any improvement or deterioration it is difficult to tie a particular result to a Threshold Program intervention. – OCMA and MSI should monitor approval rates as the new information system is installed, comparing ODECMAs with and without the new system to see if a cause￾effect relationship can be established. – Indicator 1.4, like Indicators 1.2 and 1.3, measures performance. Though interesting, and a SOAG indicator, as with Indicator 3 interventions of the Threshold Program to date do not explain either improvement or deterioration in the percentage of OCMA decisions that the CNM accepts. The indicator is even more remote from the milestones, since CNM activities have not yet begun. Measurement of this indicator has been an issue since the program began; a solution satisfactory to all parties may have to wait until the two institutions coordinate their information systems. – It is doubtful that the CNM registry contains only cases referred by OCMA, which would be a condition for the calculation being applied. Finally, the long period of time (up to five years) that the CNM can take to make a decision renders irrelevant the ―percentage of recommendations implemented by CNM,‖ and in itself could indicate corruption. The intent of this indicator is therefore to measure willingness to implement sanctions, with lack of – Add the time factor, in this case the number of days: ―percentage of resolutions issued by OCMA recommending sanctions that are implemented by the NMC within 180 days‖; a different number of days might be chosen but there should always be a time limit. In this way, the number of cases reported to the CNM is documented, and the CNM must then justify any lack of action on cases after the stipulated time is up. In this way, the program can use the indicator until a more comprehensive solution between the OCMA and CNM can be found. 14 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE III.3. INDICATORS 1.2–1.4: FINDINGS AND RECOMMENDATIONS RELATED TO CORRUPTION REPORT PROCESSING Findings Recommendations punishment perceived as impunity. For this reason, it is also a SOAG indicator. – Even with the expected participation of CNM in the program, it will always be difficult to evaluate this indicator because it involves two independent entities, and because one or the other would be at fault if performance is deficient, the tendency would be for each to blame the other. Government Counterpart: MININTER Implementer: ICITAP-MSI The MININTER goal to ―reduce the prevalence of bribes by citizens to the police‖ incorporates three types of activities: promote public awareness (information); promote implementation of the law (control and punishment); and reduce corruption opportunities (improve procedures that govern interactions between citizens and the police). These activities are measured by four indicators, two of perception and two of the process. Indicator 1.5: Percentage of citizens willing to file a complaint against the police (SOAG) The data source for this indicator is the baseline study of the population 18 years and older. TABLE III.4. INDICATOR 1.5: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – The indicator is formulated differently in USAID Reports and in the PMP of MSI. The report uses the indicator text; the MSI PMP specifies who submitted the complaint. – Standardize the wording of the indicator. – The question could induce a specific response. The indicator was constructed based on question 71: ―If you have contact with a policeman and believe that he is acting in a corrupt way ...‖ and the response options read by the interviewer were ―I denounce,‖ ―would definitely do it, ―rather not‖ and NS / NR. The first option, a commitment to action, could lead to induction of the response (fear of being judged may lead to choosing this option). – A change to a spontaneous response is suggested. According to CONECTA, the spontaneous recommendation was applied in the pilot but gave imprecise results. CONECTA suggests that a refinement would be to exchange or rotate the alternative responses. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 15 TABLE III.4. INDICATOR 1.5: FINDINGS AND RECOMMENDATIONS Findings Recommendations Milestones – Two milestones are repeated (police report) in Indicators 1.5 and 1.8. The milestones of the communication campaign were verified. The outreach activity was not complete. – Describing the milestones is suggested so their content can be reported (PSAs, police reports). Indicator 1.6: Number of days between the beginning of the disciplinary procedure and its resolution. (SOAG) Indicator 1.7: Percentage of disciplinary reports processed within the timeframe established by the law. At the start of the program, MSI calculated the baseline for Indicator 1.6, but after this was done, Law 29356 was enacted; accordingly, USAID and MSI decided it would be necessary to make a new estimate responding to the new legislation. Both indicators recognize that implementing regulations for the new law have not yet been issued. That is why data obtained from the baseline have not been used, which means that there are no data for these two Indicators. In the document review differences were found in the USAID and MSI phrasing; there is a lack of clarity in the definition of 1.7; and there are differences in some milestones. It is recommended that USAID consider replacing Indicator 1.7 in order to collect data on these activities pending approval of the regulations. TABLE III.5. INDICATORS 1.6–1.7: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – For Indicator 1.6, USAID states: “Number of days from the initiation of the disciplinary process to its final resolution (SOAG)‖ (USAID Report to MCC, second quarter of FY2010), and MSI states: ―Number of days required since a formal complaint has been filed (reception table) until a sanction has been decided‖ (MSI, Progress Report / Committee on Monitoring and Communications, June 16, 2010). The phrasing of MSI is clearer than that of USAID / MCC. – Change the phrasing of the indicator to ―Number of days required from the filing of a formal complaint until a sanction has been determined.‖ – An option, since ICITAP is in discussions with MININTER about this indicator, would be to return to the original SOAG indicator: ―Reduce to seven days the time required by the investigation and control offices to inform the complainant that the complaint has been accepted for investigation.‖ – The definition of Indicator 1.7 (USAID Report to MCC, second quarter of FY2010) could be more complete: It should include a specific number of days (set by the new Law 29356) and the types of cases that the indicator will cover. – Recommended phrasing: ―Percent of disciplinary reports of corruption in resource use and interaction between police and citizens that is processed in 30 days or less.‖ 16 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE III.5. INDICATORS 1.6–1.7: FINDINGS AND RECOMMENDATIONS Findings Recommendations – The two MININTER process indicators (1.6 and 1.7) are based on the new law. Since the regulations for this law have not been formalized, there is no baseline and no data for these two Indicators. When there are few indicators, they should be independent. The project includes three activities: Promote awareness of corruption, improve enforcement, and reduce opportunity. The two perception indicators (1.5 and 1.8) measure the development of citizen awareness. Indicators 1.6 and 1.7 measure successful implementation of standards and processes for complaints, investigation, and discipline. It would be good to have an indicator that directly measures the success of the activity to reduce opportunities for demands for bribes. Since Indicator 1.6 is a SOAG—it cannot be changed—but it would be possible to change 1.7 to cover reducing opportunities for corruption. – It would be advisable to change Indicator 1.7 to measure an aspect of the program that does not depend on the regulations for the new law. Plans of action implemented by police training participants and follow-up of the implementation of these plans could be an indicator: ―Number of action plans implemented by police who have participated in training.‖ Milestones – Milestones for Indicator 1.7 registered in the USAID report are different from those in the MSI report. – Coordinate the two versions. Indicator 1.8: Percentage of citizens that know the improved traffic procedures The baseline study was also the data source for this indicator. TABLE III.6. INDICATOR 1.8: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – The MCC/USAID RRT says ―Percentage of citizens that know the improved traffic procedures.‖ The MSI PMP says ―Percentage of licensed drivers who know the new rules and procedures governing interaction with a police officer.‖ This difference in formulation may have an effect. – Change the indicator to ―Percentage of licensed drivers who know the new rules and procedures governing interaction with a police officer.‖ PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 17 TABLE III.6. INDICATOR 1.8: FINDINGS AND RECOMMENDATIONS Findings Recommendations Milestones – Most of the milestones were not met because the activities were not implemented. – Implement activities. – The same milestone is repeated (police report) in Indicators 1.5 and 1.8 – Describe the milestones so that they can be reported (PSAs, police reports). Government Counterpart: OCG Implementer MSI The purpose of this set of activities is to reduce the prevalence of bribe-paying by citizens and companies in their interaction with public servants. Of the three indicators, one measures perception and two measure processes. Indicator 1.9: Percentage of citizens who are aware of the Comptroller General’s role in anticorruption activities. The data source is the baseline study. TABLE III.7. INDICATOR 1.9: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – The MCC/USAIDRRT says, ―Percentage of citizens who are aware of the Comptroller General’s role in anticorruption activities.‖ A separate formulation states, ―Percentage of citizens who are aware of the role of the OCG in anticorruption activities.‖ The two ways the indicator is formulated have different meanings. – Change the way the indicator is measured: ―Proportion of citizens who are aware of the preventive control activities of the OCG.‖ – How the question is framed can induce the response (done after an opinion about the OCG); there is no filter to know whether or not the – Insert an additional filter question immediately before question 103 on knowledge of the OCG.3 – Consider constructing the indicator in terms 3 The questionnaire (see Appendix 12) has a question that inquires into corruption and action against corruption in public institutions. Question P15 asks whether the respondent is aware of the existence of a number of public institutions, including the Comptroller General of the Republic. Then, for each institution, question P16 asks whether the respondent has had problems with corruption, P 17 asks whether in the past 12 months the person has taken action against corruption, and P18 whether the action was effective. But, for Indicator 1.9, question P103 the questionnaire asks: Do you know if the Comptroller General of the Republic has been doing some type of activity to prevent corruption or not doing anything at the preventive level? All 8,336 respondents answered this question; had there been a filter question, fewer people would have answered. 18 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE III.7. INDICATOR 1.9: FINDINGS AND RECOMMENDATIONS Findings Recommendations respondent knows about the OCG. of the percentage of reactions resulting from the interaction of questions P15 and P103, about knowledge of the OCG and awareness of control activities. – Reverse the order of the questions so that judgment comes after knowledge. Milestones – Milestones were not met because the activities were not implemented. Indicator 1.10: Time required by OCG to investigate an allegation of corruption (SOAG) Indicator 1.11: Number of ICOs processing corruption reports according to the revised OCG Guidebook TABLE III.8. INDICATORS 1.10–1.11: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – Indicator 1.10 is a performance indicator. After the first modification of its activities in March 2009, the OCG called for elimination of activities associated with this indicator and it became necessary to propose an alternative. At the same time, the indicator probably has a significant cause-effect relationship with Indicator 1.11 milestones related to strengthening the ICOs. However, Indicator 1.11 activities were postponed, putting into question the relevance of Indicator 1.10. In the briefing on 9 July, the OCG agreed to keep Indicator 1.10 and offered to provide data on processing time for strengthened ICOs covered by Indicator 1.11. – Reestablish Indicator 1.10—―Time required for OCG to investigate an allegation of corruption‖—but change the outcome from a percentage reduction to one expressed in number of days. – Expand the indicator 1.10 to two, one for ―control actions‖ and one for ―fast action.‖ – Because in April 2010 Indicator 1.11 activities were rescheduled, it is opportune to reconsider Indicator 1.10 with a few modifications of the goals, given that there are two types of complaints with different purposes and processes, control actions and fast actions. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 19 TABLE III.8. INDICATORS 1.10–1.11: FINDINGS AND RECOMMENDATIONS Findings Recommendations – Indicator 1.11 is a process indicator to ensure that the plan is being implemented and that results will be accomplished. The revised plan milestones are ambitious considering that the ICOs that will participate are located throughout the country. Under this plan, the equipment was installed during the last quarter, so that the system was functioning, but personnel had not been trained. OCG has the responsibility for carrying out the training according to the MSI training plan. – Advance the milestones associated with ICO personnel training. As they are currently projected, the indicator would not meet its objective as defined in the USAID/MCC report for the third quarter of FY2011. – Move forward the MSI training plan milestone in the RRT and then work with the OCG so that training begins before the last quarter. Indicator 1.12: Percentage of OCG’s implementable recommendations on selected internal control processes fully implemented. The indicator was eliminated at the request of the Comptroller because activities were changed. Indicator 1.13: Percentage of the processes involved in the provision of high-use and high￾corruption (HU/HC) services streamlined. The indicator was eliminated at the request of the Comptroller because activities were changed. Government Counterpart: Public Ombudsman Implementer: MSI The activities of the Ombudsman are reflected in a single indicator, 1.14: Indicator 1.14: Percentage of citizens that recognize the Ombudsman as an institution that monitors GOP anti-corruption (AC) initiatives. The baseline study is the data source is the baseline. TABLE III.9. INDICATOR 1.14: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – The MCC/USAID refers to the Peruvian ―government‖ while the Spanish-language MSI PMP uses the term ―state.‖ – Change the word ―government‖ to ―state‖ in the indicator. 20 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE III.9. INDICATOR 1.14: FINDINGS AND RECOMMENDATIONS Findings Recommendations – How a question is asked can induce the answer; this question is asked after a judgment is already made about the Public Defender (PD). There is no filter to see whether respondents do or do not know about the PD. – Ask an additional filter question immediately before questions about knowledge of the PD.4 – Alternatively, construct this indicator by crossing-referencing the answers the questions P15 (do you know the PD?) and P125 (do you know that the PD does follow￾up). – Reverse the order of the questions so that judgment is after knowledge. Milestones – The majority of the milestones were met. – The resolutions, minutes, and list of participants support the data reported. – Create a participant database. Implementer: ProEtica The goal for civil society is to ―Increase the capacity of citizens to use public information to effectively assess and monitor government transparency and accountability. The goal is supported by a single activity, fostering awareness, and measured by Indicator 1.15. For Component 1, ProEtica had the best organization and the most information available. Its data management and collaboration in sharing information were excellent ((see the information flow diagrams in Appendix 8). There is confusion resulting from the translation of the ProEtica indicator, a weak definition for actor, lack of disaggregation, and a possible lack of integrity. Since the baseline is zero and ProEtica has reported indicator data only once, there is no problem in adjusting the indicator. 4 As with Indicator 1.11 on the OCG, the baseline questionnaire includes the Ombudsman in questions P15, P16, P17, and P18. For Indicator 1.14, the question asks, ―Do you know or not know if at present the Ombudsman ... follows up the government's anticorruption activities?‖ All 8,336 respondents answered this question; if there had been a filter there would have been fewer responses. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 21 Indicator 1.15: Number of actors actively participating in the anticorruption network TABLE III.10. INDICATOR 1.15: FINDINGS AND RECOMMENDATIONS Findings Recommendations Validity – There is an error in the translation of the definition of the indicator. – Change the word ―academics‖ to ―university students‖ – The definition of an actor as a person who participates in an interactive activity once a month is too expansive. Participation in a single activity does not reflect enough of a commitment to support an ability to advise and monitor transparency and accountability. – Increase the number of activities required for active participation to two. – Define an actor as a person who attends meetings and performs such actions as public speaking or publishing opinions. – In ProEtica’s six-month report, the number of actors is not disaggregated by age, sex, and region. – Categorize actors by age, sex, and region. Integrity – The people who count the number of actors have something to gain when the numbers are positive, but given the wide geographic area and the small number of activities, it is not worth seeking an independent evaluation. GENERAL RECOMMENDATIONS ON ANTICORRUPTION INDICATORS In the RRT for the second half of FY 2010, MCC Threshold Program partners specify 19 Indicators, which are covered in this data review. However, there is some disagreement between USAID and two partners on four of the AC indicators. The PCM is currently proposing a change in Indicator 1.4, which involves both OCMA and CNM. The team evaluated the indicator as it was originally formulated. During the July 9 meeting, the OCG agreed to reestablish Indicator 1.10 and to provide data for processing time on the ICOs to receive new information systems. The team also asked that the OCG Internal Control Recommendations, Indicator 1.12, and the HU/HC percentages, Indicator 1.13) be retired. The evaluation team was asked to suggest replacements for Indicators 1.10 and 1.12. It is the evaluators’ understanding that Indicator 1.13 remains as it is established in the RRT until the MCC accepts its withdrawal. The analysis found that the immunization component information systems produced better￾quality data than the anticorruption component systems, which had some weaknesses in indicator and milestone definitions. It should be noted that in the case of OCMA (Indicators 1.1, 1.2, 1.3, and 1.4) one of the interventions is the introduction of a new information system due to data deficiencies in the current system. The data produced by the new system should be analyzed rather than the older data deemed deficient. The indicators should measure milestones—progress toward completion of activities. The new OCMA information system is now being implemented. No serious problems have been identified that would impede accomplishment of the projected results. 22 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT In the OCG (Indicators 1.9, 1.10, and 1.11) disagreements about indicators and milestones have resulted in near-paralysis of activities. Fortunately, the agreements reached in April 2010 led to adjustments to facilitate implementation. Only the programming of milestones is questionable due to the short time remaining before the program ends. To summarize weaknesses in the Indicators, particularly in Component 1: (1) for some there are no data because implementation has not begun; (2) for others the main intervention was installation of a new information system due to data deficiencies identified during the project design; (3) in other cases the first measurement of the indicator is set for the future; (4) some institutions have not accepted the indicators relevant to them; (5) some indicators and their milestones have been put on hold until new laws or regulations are issued; (6) other indicators are so independent of project interventions that neither their improvement nor their deterioration can be attributed to the program; and (7) some indicators have milestones that do not correspond to their definition. The recommendations of the report are organized by indicator and institution, but the evaluation team offers two recommendations on the program generally: 1. Add the program indicators to the Web pages of the counterpart institutions and consider the data published on the Web pages to be official data the quality (transparency) of which should be assessed. Performance indicators are what an institution uses in making decisions and they also inform the community about how the institution is performing, among them Indicators 1.2, OCMA time to process; 1.3, appeals to ODECMA; 1.4, CNM; 1.6, police processing time; 1.7,police processing within target ; 1.10, OCG processing; and 1.15, ProEtica. Implementation of the process and perception indicators could be shown on counterpart Web pages in the section where cooperating institutions are referenced (e.g., International Cooperation), in this case the MCC Threshold Program. 2. Establish a firm system for numbering indicators. Repeated numbering changes make it difficult to monitor the program. Even the MCC indicators experienced numbering changes when an indicator was eliminated. It is also important that the milestones be reported more than once during the life of the program; reporting should be against intermediate targets. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 23 IV. COMPONENT 2: IMMUNIZATIONS Government Counterpart: MoH and regional governments Implementer: Abt Associates Component 2 of the Threshold Program has three subcomponents; four indicators that track achievements have milestones against which to measure their progress. Subcomponent I, focused to increase coverage rates in rural areas, is implemented in eight priority regions5. Subcomponent II, the cold chain logistics and immunizations system, and subcomponent III, the information system introduced by the Threshold Program, are applied in 17 regions6. The MOH plans to extend the system nationally. ANALYSIS OF THE DEMOGRAPHIC AND HEALTH SURVEY AND BASELINE STUDY Baselines for Component 2 indicators were established using both primary and secondary information. Sources of secondary information were: Demographic and Health Survey (DHS) 2000–2007 conducted by the National Institute of Statistics and Information. From 1986 to 2000, Peru conducted a DHS every five years. Since 2003 Peru has carried out ―continuous‖ surveys. In calculating immunization coverage, the DHS collects information from the child’s vaccination card and data provided by the mother. Coverage is reported for DPT (per dose) and measles vaccinations of children 18– 29 months of age. The Analytical Reports on Immunization and to a lesser extent reports generated by the Health Information System (HIS) provide basic data. The period analyzed was calendar year 2008, and for six of the eight regions prioritized information is available for each facility (for Ica and Amazonas, it has only been possible to obtain regional information). Sources of primary information used in establishing Immunizations Component baselines were: The eight-region Diagnostic Study National Immunization Strategy by Abt Associates, which covered the regions the Threshold Program prioritized for increased immunization coverage. The study had one sample of 126 health facilities and a second sample of 32 administrative units of Regional Health Directorates (DIRESAs) as well as the health networks’ chiefs in the eight regions. Instituto Cuanto provided sample frame design, size, and distribution for its study of 1,278 health facilities in the regions comprising the Continuous ENDES 2000–2006. The sample was probabilistic, cluster and multistage. The nine-region Diagnostic Study National Immunization Strategy by Abt Associates, which covered nine regions, additional to the eight prioritized, included in the cold chain and logistics subcomponent of the Threshold Program. The study had one sample of 111 health facilities and a second sample of 14 DIRESA administrative units and health networks chiefs in the nine regions. 5 Amazonas, Apurímac, Ayacucho, Cusco, Huancavelica, Huánuco, Ica and Puno. 6 Amazonas, Ancash, Apurímac, Ayacucho, Cajamarca, Cusco, Huancavelica, Huánuco, Junín, La Libertad, Loreto, Madre de Dios, Pasco, Puno, San Martín and Ucayali. 24 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT A second study used the Instituto Cuanto sample frame design, size, and distribution to survey 1,316 health establishments in districts in the remaining nine regions that applied the Continuous ENDES 2000-2006. The sample was probabilistic, cluster and multistage. Separate questionnaires were drafted for health establishments and for administrative units. ANALYSIS OF INDICATORS Indicator 2.1: Coverage of measles (provided in Peru through a measles, mumps and rubella vaccine (MMR)) and third dose of diphtheria, pertussis and tetanus (DPT3) vaccine in eight priority regions This indicator has achieved the milestones up to the first quarter of FY 2010, which are mainly associated with operation of the IBs, their budgeted programming, and the development of regional immunization plans. Peruvian observers commented that because of budget constraints the accomplishments and operations of the IBs are unlikely to be permanent. For example, in the Ayacucho DIRESA, since May of the 13 IBs, four had ceased to function. It is imperative that as decentralization moves forward, DIRESAs take over monitoring of their own budget activities. These range from budget programming according to need, follow-up of budget execution, and establishing regular coordination meetings with the regional government. If necessary, DIRESAs might consider restricting resource use to scheduled activities in order to avoid their being directed to unscheduled activities that are only temporary priorities. The IB budget for 2011 was prepared jointly by the executive units of regional DIRESAs with technical support from Abt and points to their actions being incorporated as part of budget results, pending approval by the Ministry of Economy and Finance (MEF) (scheduled for the third quarter of 2010). Incorporation of the IBs into DIRESA operating budgets represents a significant step toward sustainable funding. TABLE IV.1. INDICATOR 2.1: FINDINGS AND RECOMMENDATIONS Findings Recommendations – The indicator meets the characteristics of validity, reliability, timeliness, accuracy, and completeness. It uses the continuous DHS, whose frequency is annual, as well as other sources: Analytical Immunization (secondary source) and survey data (primary source), reporting DPT3 coverage of children younger than one and SPR in children under two. – In its periodic reports, Abt should report not just the results of the DHS, but also data from other sources (survey and data￾DIRESA OEI). – Each of the eight prioritized regions has identical indicators. . – The indicators for both priority and nonpriority regions should be comparable but allow for differences in timing Milestones – The milestone was met by IBs operating in the corresponding period; however, in Ayacucho 4 of the 13 teams that were operating through April 2010 stopped working in May due to lack of funding. – Continue following up activities using the program monitoring and evaluation staff, so that any implementation problem is reported immediately to the PCM and MOH. They can then coordinate with PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 25 TABLE IV.1. INDICATOR 2.1: FINDINGS AND RECOMMENDATIONS Findings Recommendations Some of the remaining teams are understaffed due to high turnover. regional or other relevant bodies to make decisions on solutions. – In preparing the IB budget for 2011, corresponding to the milestone for the third quarter FY2010, it was not possible to verify that regional government teams participated in an integral way; for instance, in Ayacucho, the Assistant Manager for Social Development who was interviewed was unaware of the 2011 schedule. – Involve regional governments in formulating the budget to underscore the importance of including IBs for dispersed populations in the health system and generate commitments. – Encourage the participation of representatives of all areas having to do with health so that the turnover of authorities does not cause budget reallocations or other problems for implementation of activities. Indicator 2.2: Percentage of health establishments with adequate vaccine supply This indicator meets the characteristics of validity, reliability, timeliness, accuracy, and completeness. Progress in meeting the indicator verifies progress on milestones. This indicator has achieved the milestones scheduled for the first quarter of FY2010, in terms of cold chain equipment and it has been incorporated into the Management and Administrative Information System (SIGA, Sistema de Informacion Gerencial y Administrativa) equipment registry for purchases since 2007. TABLE IV.2. INDICATOR 2.2: FINDINGS AND RECOMMENDATIONS Findings Recommendations – The baseline reported 66.93% availability of vaccines; a review of the database found an insignificant difference (67.09%).The team was told that the difference is explained by the process of reviewing data in the annual comparison. This change has already been reported in the July 7, 2010, quarterly report. – Report any variation found in the results of surveys or reports from the MoH HIS in the next report.7 Milestones – In relation to the milestone for the second quarter of FY2009, equipment acquired since 2007 has been incorporated into the SIGA registry. Inclusion of older equipment is in process. – Follow up in the regions to ensure that regional coordinators for the program continue to incorporate SIGA equipment assets, so that each region is not limited to its own database. 7 The implementer reports any variation; this change, which was the result of comparative analysis in the previous quarter, was reported in the July 7, 2010 quarterly report. 26 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE IV.2. INDICATOR 2.2: FINDINGS AND RECOMMENDATIONS Findings Recommendations – While the milestone referred to the supply of immunization inputs through the MEF using Format 173-1S of OPS, for the third quarter of FY2009 period it was important to secure funding for vaccines. Vaccine availability at the national level does not ensure timely delivery to health facilities. Delays in delivery occurred during 2010.8 – Add one or more milestones that suggest not only the necessary means of purchasing vaccines but also compliance with delivery, storage, and distribution standards. – Provide technical assistance (TA) according to need rather than simply historical requirements. TA in the process of storage and distribution of vaccine (SISMED (Integrated supplies for Medical logistics) / DARES (Office of strategic Health Resources), should include regional and national meetings between the SISMED / DARES logistics teams and Health Immunization Strategy teams. Indicator 2.3: Time for health centers to receive information from IBs This indicator relates to the operation of the IB and reporting information as well as to the launch of IIIS in the regions. The purchasing of personal digital assistants (PDA) has begun; PDAs will reduce the time it takes to report IB data to the health centers especially once the IBs work with the unique IB-DNTP codes issued by the MoH General Office of Human Health and its units. This will unify the registration of service provided by the IBs. Regarding the milestone ―fully staffed and trained IBs,‖ it should be noted that constant staff turnover causes regular understaffing, as was the case of Ayacucho during the evaluation team field visit. Similarly, there was a decrease in the time for receiving data from the IBs. It may have been that after training IB members began at once to document interventions at the point of service rather than later, thereby speeding up transmission of information to the health facility. It may also be that during the survey limitations in understanding of both definitions and explicit questions resulted in distortions in measurement. The annual comparative analysis of establishments conducted by the Threshold Program has improved evaluation of the indicator. The pilot for IIIS was successful, use of IIIS is being phased into health facilities in the regions, and the IIIS committee board has agreed on board member responsibilities for activities and standards. Until there is a formalized directive making the IIIS official, there are cases like Amazonas, where a memorandum to establish the HIS is the sole source of information about the DIRESA’s sphere of activity. 8 Since the third quarter of 2009, Abt has monitored the availability of vaccines in the regions. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 27 TABLE IV.3. INDICATOR 2.3: FINDINGS AND RECOMMENDATIONS Findings Recommendations – The indicator meets the characteristics of validity, reliability, timeliness, accuracy, and completeness. The definition is ―Average time lapsed between receipt of data by the IBs to when the same data are entered into the record system in the health facilities in the 17 regions with IBs.‖Different IBs may report data on different schedules, such as at the time of service, at the end of the day, or when the unit returns from the field. This makes time comparisons difficult. – Clarify the indicator’s definition of, e.g., ―average time elapsed‖ between when an IB performs an intervention and when the resulting data are entered into record systems in the 17 regions with IBs. – Establish a standardized reporting protocol for all IBs to follow. This will facilitate reporting without any interpretation by the implementer or the establishment itself responsible for answering the survey or reviewing the information. Milestones – The technical specifications of the PDAs were set by the MOH and approved by USAID. Still pending are software development and installation. – Expand the milestone to cover development of software, equipment delivery, and training, which will ensure that the equipment is used appropriately. – The milestone for the first quarter of FY 2010 refers to the implementation of IIIS in 10 regions, but the definition specifies only two, Amazonas and Ayacucho. The milestone was appropriate at the beginning of implementation, but since the implementation process is continuous, the definition may need adjustment. Since the IIIS process is continuous, effectively this milestone was establish at the beginning of the project – Determine the correlation between the planned milestone and specificity of its definition. The definition needs to be appropriate to current conditions. – The milestone for the third quarter of FY2010 refers to 145 IBs that have a complete and fully trained staff but does not cover equipment, a program activity that should be highlighted because it enhances the quality of care. Equipment has been distributed to the DIRESAs but in Ayacucho and other regions has still not been distributed to the IBs. In Amazonas equipment was not distributed until the time of the evaluation. – Incorporate in the milestone not only the number of staff and their training, but also that they have equipment to provide quality care. – Continue follow-up and quarterly reporting on the number of operational IBs. – Continue follow-up of delivery of equipment purchased. 28 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT TABLE IV.3. INDICATOR 2.3: FINDINGS AND RECOMMENDATIONS Findings Recommendations – Regarding the milestone for the third quarter for regions that are implementing the IIIS, there are DIRESAs, as in Amazonas, which despite reporting progress in scaling up IIIS have issued a memorandum establishing the HIS as the only information system. – Sustain the results obtained through coordination with the Regional Health Boards and Decentralized Offices of the SIS program to ensure that the scaling-up process is not stopped. – Both findings and recommendations confirm the importance of requiring approval of the directive that establishes the IIIS as an official information source of the MOH. Indicator 2.4: Registry errors in data reported to health regional offices (DIRESAs) The MOH Technical Commission approved the pilot of the IIIS that is now being scaled up in the regions. Commission orders were issued for establishing technical committees and for progress reports. The IIIS has facilitated the Immunization Health Strategy team that monitors children vaccinated under the nominal record system in priority and other regions that had been using unique systems. IIIS also eliminates the double-counting of immunized children. There remains the important issue of adopting a national IIIS directive to formalize implementation throughout the country and thereby assure its sustainability after the Threshold Program ends. Close coordination is needed between the Comprehensive Health Insurance (SIS) and the General Office of Statistics and Informatics (GOSI), since the GOSI is directly responsible for all health information systems. Without approval of an IIIS directive, regions that have already been working with the IIIS could stop doing so in such circumstances as a change of government. Staff training is directly related to the directive. Until it is approved, training cannot be conducted, delaying achievement of goals. It is important to note that since the intervention, the number of errors reported in the logs has decreased. However, the amount of reduction varies between units, which may be due to high turnover of staff and insufficient training for new staff. TABLE IV.4. INDICATOR 2.4: FINDINGS AND RECOMMENDATIONS Findings Recommendations – The indicator meets the characteristics of validity, reliability, timeliness, accuracy and completeness. The definition ―Average time lapsed between receipt of data by the IB’s to when the same data is entered into the record system in the health facilities in the 17 regions with IBs.” Different Brigades may report data on a differing time scales. For example, at the time of service, at the end of the day, or return of the BI from the field. This makes time comparisons difficult. – Specify the origin of the data in the definition—daily vaccination records compared with the consolidated record— so it is clear that the error log report is directly related to the information system that has been used nationally and not the IIIS being implemented. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 29 TABLE IV.4. INDICATOR 2.4: FINDINGS AND RECOMMENDATIONS Findings Recommendations Milestones – The wording of the milestone for the third quarter of FY2009 is related to approval of the IIIS, but the definition has to do with its launch in three regions: Amazonas, Ayacucho, and much of Lima. – Formulate a direct relationship between the milestone and its definition so that achievements are reported properly. – Regarding the milestone for the first quarter of FY2010, Trainers trained in IIIS, reportedly it has not yet been achieved , but training coordinators of ESNI-ODSIS staff in SIS operational processes and regulation has facilitated scaling up nationwide. – Start the training, which will help institutionalize the IIIS since staff can then make immediate use of the system. OBSERVATIONS AND CONCLUSIONS Component 2 of the Threshold Program established its baseline in 17 regions (eight initially, nine added later) by gathering primary information through a diagnostic survey of the National Immunization Health Strategy and secondary information from the DHS and the MOH information system (central and regional). The MOH successfully implemented the IIIS pilot and is beginning to scale up nationally; this allows ESNI to follow up vaccinated children, making lost vaccination opportunities a thing of the past. The IBs have a 2010 budget. The MOH wrote the budget for 2011, purchased equipment, and assigned budget actions. It is still important, however, to press for actions between DIRESA, GOSI, and ESNI-General Directorate of People’s Health (DGSP), so that long-term stability will be achieved. SIGA has incorporated equipment acquired after 2007. Older equipment needs to be cataloged a task that will require a great effort by the implementer and the MOH. The mismatch between some milestones and definitions makes it impossible to determine the full scope of program activities. Additionally, some of the milestones are limited to a specific activity, which prevents successful display of a set of actions performed by the program. As a result of political or budgetary factors, milestones and achievements do not always remain viable, suggesting a need for continuous follow-up and analysis by the implementer. For reporting purposes the program collects information from a variety of sources, but in some cases the results from only one source are reported. Reporting results from different sources helps provide a broader view of results. An adequate supply of vaccines depends not only on applying inputs for immunization; the process also has to do with programming, application, delivery, storage, and distribution, all of which depend on the interaction between various agencies. This set of factors should be considered in thinking about the program and its achievements. The IIIS DIRESA is being scaled up, but adoption of the lllS directive should be cited as a milestone to consolidate it as a national system. 30 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 31 V. MID-TERM ASSESSMENT OF THE THRESHOLD PROGRAM, JUNE–JULY 2010 EXECUTIVE SUMMARY In June and July 2010 the CAMRIS evaluation team conducted a DQR and a mid-term assessment of the MCC Peru Threshold Program. For the assessment the evaluation team interviewed 115 people; reviewed documents from MCC, USAID, implementing institutions, and government counterpart organizations, and made field visits to Ica, Ayacucho, and Amazonas. Second interviews, sometimes by telephone, were conducted when clarification or additional information was needed. The Threshold Program consists of Components1, Anticorruption, and 2, Immunizations. Component 1 has four areas of intervention: training, computerized information systems and manual systems and procedures, communication campaigns, and citizen action. Component 2 has three subcomponents: increased immunization coverage in selected regions, improving the IIIS, and strengthening immunization cold chains and logistics management in the 17 regions with mobile medical teams (IBs). This assessment addresses findings, conclusions, and recommendations for each component, intervention area, and counterpart organization. It also provides some cross- cutting recommendations. The assessment recognized the limitations imposed by the short amount of time remaining for the program—eight months for Component 1 and four months for Component 2. The evaluation team believes that the program goals were too complicated and ambitious for a two￾year project, given the number of implementers and counterparts related to Component 1 and the number of systems with which Component 2 had to deal. The activities related to information systems and procedures are generally moving forward and the counterparts are satisfied with the work, so there is a strong probability that the counterpart institutions will institutionalize and sustain the improvements. However, some information systems have been delayed by institutional processes for approving actions. Moreover, Component 1 in particular has experienced changes in leadership and strategies, a lack of implementer responsiveness at times, and procurement delays tied to USAID procedures. The Threshold Program has produced considerable anticorruption communication and training that has been well received, especially by GOP counterparts, but there are still training and communication improvements that could be made in the time remaining that would give beneficiaries more useful information. A challenge for government counterparts in the transition is that they must perform training activities and communications with a much more limited budget than they currently have. Also, in general Component 2 is on schedule, except for some observations noted below. In particular, MOH approval of a comprehensive directive will allow for national implementation of the IIIS. The evaluation team did identify some weaknesses in Component 1 and made recommendations for strengthening it within the program time remaining. A challenge for government counterparts is the transition from current spending to much lower funding for training and communications activities as the program ends. Finally, the sector most likely to influence social change by monitoring government services is civil society, yet the proportion of program funding for civil society is small. Activities for Component 2 are generally on schedule, but some observations are discussed below. 32 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Because the numbers of field visits and interviews were limited, the findings, conclusions, and recommendations are not necessarily representative of the program as a whole. Recommendations for the Final Year 1. MSI should work with OCMA, OCG, and the Ombudsman to ensure that resources for the development and sustainability of MSI activities that need to be continued after the Threshold Program ends are provided for in 2012 institutional budgets, even though the amounts budgeted may be less than those available through the program. MSI should work with partners to draft sustainability strategies and adjust planned activities to provide for a gradual transition. This should not affect costs and fixed investments (e.g., purchase of equipment), which must be accelerated until the program ends. 2. OCMA and MSI should investigate options for the working groups. Because working groups are difficult to implement and their sustainability is limited, other options might be to hire law students to visit outlying areas, use existing networks for advocacy, and adopt strategies such as booths at fairs and markets that would attract the relatively few people having judicial system needs at any one time. All options also require the support of written instructive materials (see the section on training below). 3. MSI should strengthen the OCMA computer technology unit with TA and training to ensure that the new information system is used properly. With TA from MSI, OCMA should draw up a plan to reinforce the Information Technology (IT) unit with TA and training to maximize its capacity once the Threshold Program ends. 4. USAID should coordinate more closely with other donors interested in diminishing corruption. USAID has taken the lead in the anticorruption subgroup of donors, but it has a reputation for being reticent about its plans and priorities. Its current activity in drafting strategic plans for the next five years could be an opportunity to expand coordination. 5. MSI and its counterparts should review the USAID / GOP agreement to focus better on the issues. There is a perception among institutions with anticorruption interests that the content of the campaigns so far has been geared to position the counterpart institutions rather than to create awareness among citizens and institutions of the problems with and impact of corruption. 6. ICITAP should explore other opportunities for offering TA because of implementation problems with MININTER. Possibilities are to provide TA in police school curricula on ethics and corruption and to carry out an opinion poll on what the police need to combat corruption, with the results providing a basis for future action. 7. Abt, in coordination with the MoH, could work with the DIRESAs to ensure that resources for the IBs are incorporated into regional budgets. Within the GOP budget process, there is uncertainty about budget preparation at the DIRESA level. It is therefore important to make sure that IBs are in the budget at a high account level and not relegated to an easily deleted subcategory. PURPOSE OF THE ASSESSMENT USAID requested that the mid-term assessment focus on the sustainability of program activities and results and on opportunities for integration with other activities in the future (see the Scope of Work [SOW], Appendix 10, p. 12). The assessment questions are thus concerned with the following: The degree of GOP commitment PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 33 Opportunities to increase the impact of the Threshold Program Specific actions that the GOP, implementers, or USAID might undertake to maximize the results and increase sustainability Activities that should continue Effectiveness of the anticorruption communication campaign Potential USAID engagement in anticorruption and immunization activities. METHODOLOGY Information for both the DQR and the assessment was gathered simultaneously using question guides (see Appendices 3, 4, and 5). These guides were also used for all persons interviewed during the field visits to Ica, Ayacucho, and Amazonas. More than 115 people were interviewed (see Appendix 7 for a list). The preliminary DQR report was submitted after the results were presented on July 9, 2010, to USAID and representatives of counterparts and implementers. Thereafter the evaluation team conducted a new round of interviews with representatives of international donors, and nongovernmental organizations (NGOs) working in the anticorruption field, and with USAID projects. Finally, the team met to discuss findings and seek consensus on conclusions and recommendations. Limitations of the Assessment: The three areas that USAID specified for visits can be characterized as being in the initial stages of a program or having unique implementation problems; they do not necessarily represent the full scope of program intervention. COMPONENT 1. ANTICORRUPTION The Peruvian Government’s Commitment ANTICORRUPTION POLICIES The commitment of the GOP to combating corruption was boosted significantly after the 1990s when corruption had reached unsuspectedly high levels. The return to democracy after Alberto Fujimori’s 10-year administration (1990–2000), which was generally considered to be centralized, authoritarian, and corrupt, brought with it a new perspective on the relationship between the State and the people. State reforms began with two major events: the decentralization of central government functions to regional governments, and the reorganization of the State apparatus. In 2002 a national accord for governance was promoted in which all political parties and associations reached consensus on 30 State policies for the long term. The policies of this National Accord remain in force today. State Policy 26 is the Promotion of ethics and transparency and the eradication of corruption, money laundering, tax evasion, and contraband in all its forms, in accordance with which the State proposes to watch over the responsible and transparent performance of the public sector, encourage citizens to be vigilant of public administration, and reinforce the independence of the National Control System. Progress was made during the Alejandro Toledo administration, which enacted several laws to prevent and punish acts of corruption by public officers; among them are a) Law 27806, the Law of Transparency and Access to Public Information (2002), which establishes the State’s obligation to encourage transparency, publish and grant State 34 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT information to all citizens, and obligate all State entities to operate transparent Web sites on which are published documents on administration, budgets, procurement, and activities of public officers. b) Law 27482 (2001), which regulates the publication of sworn statements by public officers. c) Law 27815, the Law of the Code of Ethics in Public Office (2005), which establishes goals, principles, obligations and prohibitions governing the exercise of public office. The Toledo administration also a) Created the Financial Intelligence Unit. b) Issued directives governing the neutrality and transparency of public institutions and officers in the 2006 general elections. c) Set up codes of ethics for each public institution. The government of President Alan Garcia, under the direction of the PCM, has continued the efforts to fight corruption initiated by President Toledo. In October 2007 the GOP created the National Anticorruption Office under the PCM to replace the deactivated National Anticorruption Council that had been part of the judiciary. That office was in turn deactivated in August 2008. At the end of 2008, with the change of Prime Minister because of a scandal over influence trafficking, the GOP drafted a National Plan to Combat Corruption on the basis of an earlier plan produced in 2006. The National Plan has four objectives: (1) strengthen the joint system to combat corruption; (2) institutionalize throughout government public administration practices of good governance, ethics, transparency, and combating corruption; (3) integrate an effective and comprehensive judicial strategy against corruption; and (4) encourage practices and conduct in the private business sector to combat corruption. However, this plan still needs supporting regulations to make it binding. In 2009 a Multisector Working Group was established (Ministry Resolution Nº044-2009-PCM) to drive implementation of the National Plan to Combat Corruption. During the first months of 2010 the GOP also created a high-level Anticorruption Commission, presided over by the Prime Minister with representatives from a number of sectors. This commission is in charge of proposing performance measures and competency goals for government agencies in order to execute the National Plan, monitor its progress, and coordinate its execution with the legislative branch, the judiciary, autonomous constitutional institutions, political parties, civil society organizations, business associations, and professional colleges. To date the commission’s decisions have not yet been made public. In July 2010, Law 29542 was enacted to protect whistleblowers in the administrative spheres and promote effective collaboration in the criminal sphere. Its objective is to protect and grant benefits to public officers and servants or citizens who report well-supported information of illegal acts carried out by institutions. Institutional Changes to Fight Corruption Judiciary: Because of the corruption attributed to the Fujimori administration, the Government created special criminal courts to try State corruption cases. In 2010, following the corruption scandals that arose in Fujimori’s government, the President of the Supreme Court presented a bill to Congress to create a National Anticorruption Criminal Court that would speed up court procedures. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 35 Previously, in 2006, a National Anticorruption Council was created as a unit of the Ministry of Justice. The council had permanent status. Its objective was to attack corruption head-on via prevention and shaping values, training public officers and servants, and educating society about ethics and morality. This commission was replaced by the National Anticorruption Office subject to the Presidency of the Council of Ministers. Comptroller General of the Republic: In the previous government, control systems were strengthened by decentralizing the OCG. In 2010 a process was begun to slowly transfer the IOCs from their Institutions to the OCG, thus preventing an institutional dependency that could undermine the objectivity of the control actions. In 2006 internal control norms had been approved to improve public administration; in 2009, the Internal Control System was established and applied progressively, taking into account the functions of each institutions and the availability of funding. In July 2010 Law 29555 was enacted to implement progressively the positions and budgets of the OCG institutional control entities. This will consolidate the independence and autonomy of the supervision of public resources. Designation of positions and budgets to contract auditors will begin in 2011. Public Ombudsman: The Program for Public Ethics and Prevention of Corruption was implemented in 2010 with staff and budget allocated. A document was published this year on the role of the Ombudsman, for ethics and anticorruption. Citizens’ perception: Despite significant advances, citizens continue to perceive that corruption proceeds unpunished: 55% of citizens rank corruption as the country’s principal problem and believe that it hampers development. ProEtica’s Anticorruption Poll found that citizens consider the Fujimori government to have been the most corrupt, followed by President Garcia’s first government, with the Toledo and Belaunde administrations sharing third place. Alan Garcia’s present government is considered more corrupt than those of the previous governments and those polled consider that the Aprista government shows no interest or is very disinterested in fighting corruption. The perception of corruption applies to the judiciary and the Congress as well as the executive branch. The perception of impunity has been exacerbated by the punishments meted out in corruption cases, involving high government officials and leaders of the ruling party charged with influence peddling and corruption generally. The perception is reinforced by the fact that the judiciary and the public prosecutor have been seen as being slow to investigate and punish acts of corruption. State institutions are seen by 90 percent of those surveyed as lacking commitment to fight corruption. People interviewed for this assessment expressed similar opinions. Conclusions Measures taken to (a) punish corrupt officials; (b) prevent acts of corruption; and (c) involve citizens in monitoring and preventing acts of corruption seem to have been ineffective, although since the Fujimori era, the State seems to have taken major steps to deal with corruption. New laws, regulations, and institutional restructuring and pledges from political parties are positive initiatives but the results have been far less than the public expects. The public perception is that corruption continues unabated and with impunity. Current scandals have made people even more pessimistic. The situation could even undermine Threshold Program efforts if the public perceives the communications campaign to be a slick government effort to convince them that something is being done about corruption while business continues as usual. There is little reason to believe that public perception will improve until the public sees that impunity is a thing of the past and that sanctions commensurate to the infraction are applied impartially. 36 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT To begin to turn the situation around, Donors and others involved should encourage increased participation by civil society and the private sector in monitoring and preventing corruption. Citizen participation and investigative journalism are important in making government institutions work and complement information provided by public entities. State institutions should continue the efforts encouraged by the Program to modernize their operations and make them more efficient and transparent. This includes accelerating and mechanizing the complaint and reporting processes to lessen opportunities for corruption and prove to claimants that their cases are being taken seriously. State institutions should inform the public of what they are accomplishing: number of cases resolved, quantity of money recovered, and the number and severity of punitive measures applied. This information should be published on institutional Web sites and through mass communications media. The World Bank and the Inter-American Development Bank (IDB) offer anticorruption campaigns that the GOP should investigate to determine if they are appropriate to the situation in Peru. Findings, Conclusions, and Recommendations This section is organized according to the principal activities financed by the program. TRAINING Counterpart: OCMA / ODECMAs Implementer: MSI Activity: OCMA Judicial Transparency Workshops Findings: The OCMA training workshops are focused on informing the population about the magistrates’ control mechanisms, procedures for filing a report, and citizen participation. MSI developed materials and models for OCMA and the ODECMAs to use in training about how to record complaints against the public administration. OCMA is carrying out the workshops in all 29 judicial districts. By the end of 2009 it had held 32 workshops in which 1,812 people participated. The participants are leaders of grassroots organizations, staff of professional colleges, and public servants. For the second year a similar number of workshops are planned with the same ODECMAs but with more and different people. Workshops are held on Saturdays (8:30am -6pm) and are preceded by a two-hour master conference on Friday. During the Saturday sessions, MSI conducts four hours of training followed by an afternoon session on how to set up a working group. The program goal is to train 3,050 people. The principal product of the workshops is the working groups comprised of representatives of professional colleges, business and workers associations, organizations and associations connected to the judicial sector, law faculties, the press (print, radio, and television), grassroots organizations, and other interested parties. The objective of the working groups is to share what is learned in the workshops with the larger community. The role of the MSI is to provide training; whether participants decide to participate in working group is not its responsibility. ODECMA provides support to the working groups. If necessary, OCMA’s Visits and Prevention Unit will provide direct support. Conclusions: The sustainability of the working groups is discussed in detail below; these conclusions are limited to the training workshops. The training model and materials designed by MSI were modified and improved as experience with the workshops grew. The current PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 37 methodology is case analysis, about which participants interviewed are positive. Through the workshops, participants learned how to initiate corruption complaints and were motivated to share their knowledge with the community at large. However, the evaluation team did not carry out an assessment of the workshops with participants and there has been no follow-up of post-workshop activities. The opinion of OCMA staff is that the ODECMA in Piura had the best workshop results, followed by Ayacucho. No working group met after the workshop in Chachapoyas, Amazones. In terms of sustaining a workshop program, a positive factor is OCMA’s commitment and assignment of this responsibility to the Visits and Prevention Unit. However, OCMA does not have enough staff or resources to continue with the training, and the assignment of responsibility may be limited by the end of the program. The assessment team found during visits to Ica and Chachapoyas that to carry out the training workshops ODECMA magistrates subsidize the cost of photocopies and a coffee break with personal funds. In Ayacucho ODECMA uses its own resources to execute the educational activities contemplated by the program. While the ODECMAs are responsible for replicating the training provided by MSI, MSI creates a psychological barrier because its staff arrive with a sophisticated team of trainers and elaborate manuals, stay at the best hotels, partially cover the costs of participants, and then leave with the message that the ODECMA should replicate this. The Threshold Program schedule sets up two workshops per judicial district. The cost of maintaining such a schedule once the program ends could be significant for OCMA and the ODECMAs until the working groups become institutionalized. The costs include the time invested by OCMA and ODECMA staff, transport to and within regions, workshop materials, and the transportation costs of participants. Because the Awareness Raising Program carried out by the working groups is restricted to a specific group within the public, their coverage is limited. In any case, the principal target audience should be members of the public who use the services of the judiciary, who may not necessarily be reached through the working groups. Recommendations: MSI has made it very clear that it is responsible only for the training. However, if the training is to result in the continuous transmission of information, some modifications are required. Before the program ends, MSI should develop a follow-up strategy to not only monitor the workshops and their results but also to assess the impact of the working groups. It should also formulate a model that technically and financially OCMA and ODECAMA can follow. Information on judicial branch mechanisms for control and complaints should be addressed principally to the users of such services. The options the program could study for disseminating the information are infinite and beyond the scope of this assessment, but a broad range of examples is provided in the section on Citizen Action, Working Groups. Counterpart: MININTER—Peruvian National Police (PNP) Implementer: ICITAP Activity: Training for the police on ethics (24 workshops in 8 focus regions). Findings: ICITAP conducted two workshops for 52 people, with the objective of educating trainers, who are currently replicating the training in regional workshops. The objective of the workshop is to revise personal values and the values of institutions in order to attain greater concordance between them and help form better people. The workshop dynamics were much more personal than the usual institutional training, and the few participants the evaluation team was able to interview (in Lima and Ayacucho) had positive opinions about this methodology. The training was directed to officers and subordinate personnel. There were problems in reaching the planned number of participants. The candidate vetting process was long and complex. Some of the police whose participation was approved did not 38 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT attend the workshop; others who were not approved came to the workshop although they were not accepted. In Ayacucho, the training date was brought forward at the last minute, which made it impossible for half of the approved participants to attend. Although one person interviewed stated he was very motivated by the training, when asked if the course had included a plan to replicate or disseminate the information within his work place, he said it had not, but he would like to replicate the course with his colleagues at work and even at the university, though he did not have a structured plan for how to do it. He had some ideas, for example, talking about the issue at the time of roll call or in other workplace meetings, but he did not know how to organize his ideas. Because the methodology used in the ethics training is different from the usual type of police training, ICITAP is coordinating so that the methodology can be incorporated into the curriculum at the Police School and thus made sustainable. Since it is recognized that curricula must be continually revised, ICITAP is backing formation of a unit to renew the curricula of courses and continuing education for the national police force. Conclusions: Although the training designers intended to change attitudes in the participants rather than bring about specific action, the investment in training should include replication plans or, better still, action plans to combat corruption within the institution and go beyond the goal that participants ―become a better human being‖. The fact that the number of people trained is limited makes it difficult to generate institutional change, establish a system of replications, or install this training model and its subject matter into the curriculum of the national police force. The assessment team found that the national police publicize new regulations on punishments during daily roll calls but do not have photocopies of the regulations to distribute or methodologies or equipment to analyze concrete situations of corruption. The persons interviewed consider it necessary to publicize the issue of values and anticorruption policies and introduce these themes in the National Police School, from which approximately 6,000 members graduate every year. Recommendations: Revise the contents of the training program so that participants can apply the ethics in their day-to-day tasks. The training should include development of an action plan to combat corruption and a replication plan, including periodical meetings to monitor progress against the plans. For courses still to be held, a larger number of people should be invited. Vetting should not result in fewer attending than the capacity of the program. The ethics course and its methodology should be incorporated into the curriculum of the National Police School; to do so within the framework of decentralization, a practice and awareness workshop could be given for those responsible for police schools in the regions. Counterparts: Public Ombudsman’s Office Implementer: MSI Activity: Training in public ethics Findings: The Ombudsman’s Office, through Antonio Ruiz de Montoya University, is offering workshops on public ethics for leaders in civil society and public servants in five regions. More than 500 people have received the training. he Ombudsman’s Office also has an Ombudsman School that, through distance learning modules, gives ethics courses to the institution’s personnel. Conclusions: Threshold Program financing facilitates the offering of these training activities. Were it not for these funds, the Ombudsman’s Office would have to seek support from other donors. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 39 Recommendations: In order to continue the training program, the Ombudsman’s Office should include in its training budget specific line items to support continuing the training sponsored by the program. Implementer: ProEtica Activity: Anticorruption schools in two regions to train members of their networks. Findings: ProEtica’s regional baseline studies showed that ―citizens were discouraged and demoralized about the subject of corruption.‖ This is the source of ProEtica’s philosophy of giving constant attention to its networks to vitalize them, through training, distribution of materials, and contact with monitors who link the regional networks with the national office. Trainings called anticorruption schools were held in August and September 2009 and 105 people attended. In the third quarter of FY2010, the networks were invited to regional meetings to reinforce the initial training. The ProEtica training combined theoretical classes on corruption with information on methodology, instruments, and tools to combat it. The collaborating organizations (Andean Commission of Jurists, the NGO Transparencia, and the Press and Society Institute [Instituto Prensa y Sociedad]), drafted specific manuals for the training on, e.g., ―Monitoring Justice,‖ ―Monitoring the Finances of Political Parties,‖ and ―Investigative Journalism.‖ According to the report for the second quarter of FY2010, ProEtica built a national network of 145 people who had attended at least one meeting of their regional network. During field visits the evaluation team interviewed members of the Ayacucho, Ica, and Amazonas regional networks who spoke enthusiastically of their interest in combating corruption. It should be noted that ProEtica’s activities began later than those of other Threshold Program implementers; in fact, in Amazonas three of the participants were first invited the day of the visit by the assessment team. Conclusions: The anticorruption schools combine theory and practice; the training in advocacy continually stimulates the networks by sending new materials, holding conversations with the monitors, and holding follow-up meetings. Recommendations: ProEtica’s work could be more effective if it had could leverage more funds. IPYS, a member of ProEtica, could do more in terms of promoting publicity campaigns. For example, it could offer regional courses on communication for network public relations people. ProEtica should join and help in coordinating the group of institutions with which MSI is working; ProEtica’s efforts would be even more effective if its members were in contact with other Threshold Program counterparts and implementers. It would be especially interesting to explore incorporation of members of ODECMA working groups in the ProEtica networks, and vice versa. INFORMATION SYSTEMS All the government counterparts are receiving support for their information systems. The establishment and installation of the OCMA system is more comprehensive than those of the OCG, Ombudsman’s Office or the National Police. Counterpart: OCMA/ODECMAs Implementer: MSI Activity: New OCMA/ODECMA Information System The information system now currently being installed will raise the capacity of OCMA and the ODECMAs to manage complaints so that cases can be processed more quickly and thus offer less opportunity for corruption. Once the pilot stage is concluded, the system will be extended 40 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT to 29 ODECMA offices, providing access through the OCMA Web site. Under the current system, only 10 ODECMA offices can input information into the central system; the others record their cases on Excel sheets, which are sent to OCMA to be integrated into the system. Conclusions: The new system is being installed in the OCMA Lima office, is still in its pilot phase, and is not yet connected to the Internet and the ODECMA systems. Computers have not been distributed to the ODECMAs although they have arrived at the Customs offices. OCMA is satisfied with the design of the system and its experience so far is positive. Implementation is behind schedule, but if the Threshold Program is extended, the system should be working in all the ODECMA offices once it ends. The sustainability and benefits of the new system should not be a major problem. OCMA has an experienced IT team, although they require additional TA, especially training; the institution is working actively with MSI on installation of the system. Recommendations: Complete installation of the new system as quickly as possible. If implementation does not elicit data on Threshold Program indicators, it would be important to adapt the system to collect the data. Counterpart: MININTER Implementer: ICITAP Activity: System for processing reports Findings: By agreement with the MININTER, the National Police are responsible for installing the cables required to connect Program-donated equipment. Because this has not yet been done, installation of the computers and the new data processing system has been postponed. ICITAP provided TA to improve specific National Police processes, such as (1) revision of Traffic Division procedures, which is almost concluded; (2) drawing up a flow chart of the Inspector General Office’s work process; and (3) designing a computer technology system to monitor reports of corruption. The procedures, charting of processes, and systematic processing of reports are contributions that will be sustainable to the extent that the National Police use them. Conclusions: The counterpart did not facilitate the implementation effectively. Recommendations: In seeking future support the final stage of the Threshold Program should specify the needs of the National Police in terms of combating corruption. Counterpart: OCG Implementer: MSI Activity: New Information System for the Comptroller’s Office of Internal Control, ICO. As part of the Threshold Program, this activity was intended to improve the information system of 250 ICOs by opening up access to the central system through the OCG Web site. Conclusions: As a means of minimizing corruption, the intent was to speed up the processing of reports on and mitigate opportunities for corruption. In March 2009, to mechanize a process designed by The German Society for Technical Cooperation, the OCG proposed a system, under the Threshold Program, that included equipment and trained personnel for the 250 ICOs. While the proposal was considered important, the activity was never initiated due to a lack of agreement among the several offices of the OCG. Finally, in April 2010 USAID and the PCM reached an agreement with the OCG on a new implementation strategy, which has already begun although the new proposal is not yet formally approved. The challenge now is to fulfill the plan before the program ends. The number of ICOs involved has been reduced to 94. The time was reduced due to (a) past delays; (b) coordination and articulation of the Threshold PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 41 Program’s proposals with others in the State Modernization Program (loan from the Inter￾American Development Bank); (c) the OCG’s decision to establish regional offices; and (d) the transfer of all 800 ICOs from their individual institutions to the supervision of the OCG, a complicated process that will take several years. The OCG has an ambitious, seven-year strategy that places a high priority on enhancing productivity so that it can process corruption cases expeditiously. The benefits and sustainability of the new system in the ICOs should not be a problem. The OCG has both an experienced IT team and a school for training its personnel. The only possible threat to the system would be imposition of a noncompatible information system from another donor while more ICOs are being added to the system. Recommendations: Systems should be implemented in the ICOs speedily so that the activity is completed before the program ends. MSI will design the training program for affected ICO personnel and OCG will provide it. It is important that the design of the training program be integrated into the curriculum of the OCG school to ensure that the same material is used to train personnel in other ICOs as they join the system. COMMUNICATION CAMPAIGNS Counterpart: OCMA, MININTER, OCG, and Ombudsman’s Office Implementer: MSI Activity: National and localized campaigns The perceptions of those interviewed about the communication campaigns are very divergent. Some were not aware of the communication activities (―I don’t watch television,” “I don’t listen to the radio‖). Consequently, the results of the interviews are not systematic. Findings: Nationwide campaign “Vamos a sacarnos la mugre” (“Let’s take the dirt off”). MSI hired Toronja Comunicación Integral to develop creative concepts for the Component I anticorruption communication campaign. Concepts the company came up with were evaluated in focus groups led by CONECTA Associates. The themes with the most support were ―Let’s clean up Peru‖ and ―Let’s get the dirt off.‖ For the national campaign, MSI sought consensus among the four government counterparts in order to carry out a comprehensive campaign. The Ombudsman’s Office chose to use the slogan, ―We are on your side in this fight‖; the other three of the institutions agreed on ―Let’s take the dirt off for Peru.‖ The traffic campaign. The traffic campaign emphasized nationwide radio spots. In the visual materials, a member of the National Police explained how a citizen should interact with the police. “No a la coima” (“Say no to bribes”). The campaign was implemented on the beaches using T shirts, caps, stickers, and balloons carrying with this slogan. Pro and con. The spot those interviewed most admired had a group of children reciting instances of good conduct and then asking adults why they don’t act that way. Web sites. Several people mentioned that websites are important forms of communication for an institution combating corruption. They thought that the web pages should publish the progress of complaints and ultimately the punishments so that citizens could know what cases were being resolved. ProEtica: This civil society organization designed a campaign based on the use of public resources in regional and municipal elections in 2010. The principal figure, a young girl named 42 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Lupita, scrutinizes candidates for office with a large magnifying glass, making sure that only honest and hard-working people with an authentic vocation for service are running. Although this campaign helps meet the objectives of Component 1, it is being financed by a ProEtica￾associated network rather than the Threshold Program. Conclusions: Several people interviewed, including national and regional government officers and officers from international organizations, disapproved of the campaign slogan ―Let’s take the dirt off Peru,‖ because it seems to imply that the government is completely corrupt. However, the phrase can be interpreted several ways, especially when rallying support. In its assessment CONECTA Associates found that citizens did not directly associate the phrase ―Let’s take the dirt off‖ with corruption. They only made such an association when the phrase was mentioned in the specific context. On the other hand, so far the campaigns have centered their messages more on positioning the institutions rather than educating the public (What is corruption? What is the role of the citizen faced with corruption? How should one act when faced with a case of corruption?). Also, they took no notice of the role of civil society corruption and were not explicit about the benefits of not being corrupt. Messages should seek to create awareness and explain the benefits of not being corrupt. Counterparts in the regions, civil society, and other donors interviewed though the campaigns may have been confusing to citizens because a variety of messages were publicized at the same time (the OCMA campaign, the OCG, and the Ombudsman’s Office all conducted campaigns at the same time). A major concern is that the counterparts are putting too much emphasis on using the campaigns to position their organizations and not building awareness of corruption in the institutions themselves and among citizens in general.9 While this assessment was being carried out, the Ombudsman’s Office, the OCG, and OCMA separately launched radio and television campaigns, each with a different message. The first related to corruption in the case of rights, and the other two on the role of their institutions. This is counterproductive if the intent is for the public to internalize the message. Moreover, parallel to the campaign the implementer was carrying out interviews for the second polling; the results may be affected (even induced) by the communication campaigns. Recommendations: The USAID/GOP agreement should be revised to give MSI and the counterparts better guidance about the issues in communication campaigns. Indicators and execution of the benchmarks should be published on the websites of the counterparts. Positioning the institutions may be necessary, but there are questions of whether the timing is opportune and whether the campaigns contribute to what the project is seeking to create: awareness of corruption. CITIZEN ACTION Implementer: ProEtica Activity: A nationwide network to assess and monitor corruption and transparency in regional public administration As part of the training given in the anticorruption schools, participants prepared proposals for action against corruption to be implemented in the regions with small donations ($4,000 or S/.12,000). In Ica the proposal related to reducing corruption in municipal elections; in 9 MSI, commenting on the present assessment, said that it was necessary to position the institutions because the public was unaware of their role and their existence. Although that is certainly true, the program’s objective as specified in the contract is to create citizen awareness about corruption. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 43 Ayacucho, it related to hiring school teachers without corruption; and in Amazonas related to supervision of municipal social programs. ProEtica proposed that the networks be made up of at least one representative from an NGO, one journalist, and two university students. Findings: After nine months of experience with the networks, ProEtica found that the participation of the university students was irregular given the pressure of their studies and their tendency to travel during vacation time. In Ayacucho all the members of the network belong to the sponsoring NGO. In Amazonas, three of the six members were university students who participated for the first time when the evaluators visited. In order to strengthen the networks, ProEtica has proposed for the next edition of the anticorruption school publication that the groups be made up of two NGO representatives and two journalists. The institution has an advisory council of anti-corruption experts who play a significant role in guiding the organization. However, ProEtica seems to perceive of itself as being outside the Threshold Program’s main line of action. ProEtica has received a donation from the Open Society Institute (OSI) to support the networks for an additional year. Conclusions: ProEtica has been flexible in its response to the problem of composition of the networks, immediately changing its methodology. Given that ProEtica began its work after the other counterparts and that its budget is limited, the institution was able to make up regional teams quickly (in five quarters). The networks were only recently established and depend on the participation of volunteers. If these networks are to be sustainable, it is necessary to define their functions and actions. It is not possible to estimate the impact of the networks because they have been in operation for such a short time, but the evaluation team considers that active participation of civil society contributes to the success of the Threshold Program. Recommendations: With the proposed new network structure enlarging the role of journalists, it would be useful to organize a workshop specifically for them on combating corruption using social communications media. Also, when appropriate MSI should include ProEtica in meetings program counterparts. Counterpart: OCMA/ODECMA Implementer: MSI Activity: Organization of Working Groups by the ODECMAs Findings: The working groups are a product of workshops OCMA conducts with MSI support in each of the ODECMAs. At the end of each workshop the facilitators ask participants to organize a working group to replicate what they have learned. In Ica and Amazonas, the working groups did not operate after the training. In Ica the members had other expectations of their duties (such as direct interventions in corruption cases and filing specific complaints), and in Amazonas the working group was never convened. In Ayacucho, the reason for the working group was understood to be to resolving complaints, and some members resigned when it was explained that this was not so. However, the Ayacucho working group continues to convene. There is concern in OCMA and the ODECMAs that the working groups in Ayacucho and Puno might become political, exerting pressure on ODECMA judges. This could mean that the purpose of the working group was not well explained or it was not understood. Also, participants of the working groups expressed expectations that their expenses should be covered, particularly transportation. For participants who live far away (as much as four to five hours travel time), cost is a factor that could limit their availability. Conclusions. USAID should consider other options besides working groups, though the spaces created by the working groups are sustainable with staff support and financing, especially for transportation. The percentage of the population who need to file a complaint is minimal, given 44 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT how few members of the public seek the services of the judiciary. The program should therefore focus its efforts on specific groups that use the courts to ensure that they obtain the information necessary to file a complaint instead of training a large number of people. During the rest of the program, attempts should be made to segment the population to see what groups are likely to have the greatest need and to explore how to reach people with current needs and offer them one-on-one consultation, ideally using volunteers. Representative options are: 1. Offer one-on-one guidance to persons who turn up at the Judiciary offices. Volunteers, who could include participants in the workshops, could carry this out. 2. Redecorate the ODECMA offices so they offer a professional atmosphere and foster direct contact with the public and make available promotional material, waiting areas, and offices providing privacy for consultations and complaint filing. 3. Have working group members and ODECMA representatives participate in events and informational fairs. 4. Since law students have an obligation to work in social service, have them provide legal advice in rural areas instead of bringing rural leaders to a central location. 5. Print communications on low-cost materials (for example, on newsprint) to achieve wider coverage. 6. Broadcast radio call-in programs to answer callers’ questions and offer legal advice. 7. Instead of only making presentations to civic groups, ask these groups to organize and publicize events in which legal advisors would be available to answer questions. Recommendations: Explore options, such as the fairs, that OCMA and ODECMAs could deliver more easily and at lower cost Counterpart: MININTER Implementer: ICITAP Activity: Police monitoring plan with activities in communities Findings: Community activities fall within the scope of the Family, Citizen Safety and Protection Office. This area has been forgotten and has a minimal budget. Although each police station has a division geared toward the community, there is no direct involvement with the community, so that community activities are unsustainable. Moreover, the National Police has no protocols or procedural manuals, and the lack of follow-up for actions contributes to the low execution of an already minimal budget, making it difficult for the police to benefit from the Threshold Program. It cannot be determined whether the work sessions fulfill or will fulfill their expected role. Conclusions: Counterpart involvement is key to execution and sustainability of programmed activities. Also, the Threshold Program does not seem to recognize that the National Police has other priorities. Recommendations: Due to the implementation problems with MININTER, it would be well to explore other TA possibilities for meeting MININTER’s needs. One possibility is to incorporate the issues of ethics and corruption into the Police School curriculum, and to carry out a diagnosis of what the police need to combat corruption and use the results to propose actions for the future. Opportunities to Increase Threshold Program Impact Component 1 focuses on reducing corruption in public administration and generally targets its interventions to create citizen awareness and strengthen institutions. Education and information PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 45 programs generate citizen awareness. However, program actions are still in the initial stages and alone are unlikely to generate change. Additionally, concrete actions to reduce and prevent corruption are not being tackled; instead, activities are directed to building up institutions to deal with corruption after it has occurred. The principal areas where changes should be made to improve Threshold Program impact are training, communication campaigns, and citizen action. HOW TO MAKE THE COMMUNICATIONS CAMPAIGN MORE EFFECTIVE How effective the communications campaign can be in changing attitudes and practices is limited by the short time left for the program. It is probable that the results of the survey currently underway will reflect the population’s perception of recent acts of corruption, which have been widely covered in the mass media, rather than messages absorbed from the campaigns. Current events could thus distort the results of the survey. People interviewed have conflicting opinions about whether the campaigns were informative, interesting, and attractive. They also commented that the campaigns were just publicity gimmicks that only benefited the organizations that promoted them and did not provide practical information, such as how to initiate a complaint. These opinions may not be representative, but they do justify a careful review of the messages and their reception. Those interviewed also observed that local languages were not used and messages were not adjusted to the region or location where they were broadcast. OCMA, however, attributed the significant increase in the number of complaints to the success of the campaign, which suggests that the message was effective. The fact that different messages were being promoted at the same time by OCMA, OCG, and Ombudsman may have led to a specific message not being remembered or to confusion for the public. Also, some campaigns were carried out ahead of the activities they promoted; for example, campaigns have been launched on procedures for filing complaints through OCMA/ODECMA and the IOCs, but the information systems to facilitate the complaints processes have not yet been installed. RECOMMENDATIONS FOR THE FINAL YEAR OF THE THRESHOLD PROGRAM 1. MSI should work with OCMA, OCG, and the Ombudsman to ensure that resources for sustaining MSI activities continue after the Threshold Program ends and are included in the 2012 institutional budgets, even at a reduced level. MSI should work with partners to develop sustainability strategies and adjust planned activities to provide for a gradual transition. This should not affect costs and fixed investments (e.g., purchase of equipment), which must be accelerated until the program ends. 2. OCMA and should MSI consider options for replacing the working groups because they are difficult to launch and their sustainability is limited. Options might include hiring law students to visit outlying areas, using existing networks for advocacy, and adopting strategies such as booths in fairs and markets that would reach the relatively few people who need access to the courts at any given time. All options would require supportive written instructive materials. 3. MSI should reinforce the OCMA computer technology unit with TA and training to ensure that the new information system is used properly. With TA from MSI OCMA should draft a plan to support the IT unit with TA and training to maximize its capacity once the Threshold Program ends. 4. USAID should coordinate more closely with other donors concerned about corruption. USAID is leading the anticorruption sub-group of donors but has a reputation for not being 46 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT open about its plans and priorities. Its current activity in developing strategic plans for the next five years could be an opportunity to expand coordination. 5. The USAID / GOP agreement should be reviewed by MSI and counterparts to tighten the focus of campaigns on the issues. There is a perception among institutions with anticorruption interests that the campaigns so far are geared more to positioning counterpart institutions than to creating awareness among the public of the problems associated with corruption and their impact. 6. ICITAP should explore further opportunities to offer TA because of implementation problems with MININTER. Among the possibilities are adding to police school curricula TA on ethics and corruption and offering to carry out an opinion poll on what the police need to deal with corruption. COMPONENT 2. IMMUNIZATIONS Peruvian Government Commitment and Sustainability In 2002, political parties and associations within the National Accord reached consensus on health priorities in Peru, among which were universal access to health services and promotion of food security. In July 2008 the National Accord ratified commitment to the priorities of child health and combating child malnutrition. Peru has managed to eradicate and control a series of immune-preventable diseases, thanks to the Government’s permanent commitment to guarantee a fund for immunizations and the support of cooperating institutions. This commitment supports the sustainability of actions implemented by MOH itself and the cooperating institutions, provided they are carried out in coordination with ESNI. The MOH health objectives for 2010 include combating maternal mortality, reducing infant mortality and chronic malnutrition in children under five years of age, prevention and treatment of acute respiratory and diarrheic infections, and increasing national coverage of immunizations. The budget for FY 2010 includes budgetary credits for the MOH Strategic Nutrition Program, with funds to be distributed to regional governments for administering vaccines to children under five years of age, to purchase vaccines and syringes, and to defray operating costs. Few other governments in the Americas have assigned a budgetary line item to cover the operating costs for an expanded immunization program. Peru’s commitment is a worthy example for its neighbors in the Americas. 10 The current administration has defined 12 mandatory national policies,11 such as reducing infant mortality and chronic child malnutrition. These goals were incorporated into the Multiannual Macroeconomic Framework and the Multiannual Social Framework for 2009–2011 in compliance with the Social Programs Reform Plan. The policies are monitored every six months, and the Public Budget 2009 law ratifies the goals on chronic malnutrition, maternal and neonatal health, and access to health centers, among others. To begin to reach these goals, the Peruvian Government has created a conditional cash transfer program for the poorest. This program, known by its Spanish acronym ―JUNTOS‖ (meaning ―together‖ in English), reaches those Peruvians who are poorest and most at risk. To receive modest monetary grants, participating mothers must register their children, take them for growth and development and malnutrition examinations, and enroll the children in school and ensure their attendance. Another current government activity is ―CRECER‖ (meaning ―to 10 Boletín de Inmunización–Volume XXXII, Number 1– February 2010. 11 Supreme Decree Nº 027-2007-PCM dated March 2007 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 47 grow‖ in English), a national strategy that brings together the efforts of public institutions, private entities, international cooperation and civil society in general to combat chronic malnutrition and eradicate poverty. Both JUNTOS and CRECER reflect the commitment of the GOP and civil society to the issue of children’s health, including immunizations. While JUNTOS and CRECER are administered separately from the immunization activity administered by the DIRESAs, the three programs share a target population of poor, underserved children. In view of the 2010 municipal and regional elections and the general election in 2011, it is important to implement actions that create awareness among, and advocacy by, the candidates and political parties so that they respect existing agreements and national priorities. The processes initiated by the Immunizations Component of the Threshold Program are directly related to national priorities and offer an opportunity to support State systems and strategies of the State: the SIGA logistics system, the SIS information system, and the IB teams. An important aspect of these is the joint work carried out by different parties: MEF, the DGSP, General Office of Statistics and Data Processing (OGEI), SIS, and the General Planning and Budget Office. These processes require more than two years to establish and sustain regionally and nationally. A coordinated effort needs be made now with MOH departments and regional governments to ensure that activities already begun continue, taking into account that the program is near its end, that there will be changes in regional and local government administrations at the end of 2010, and that there were will be changes in the central government late in 2011. The management, planning, and budgetary programming of the DIRESAs needs to be reinforced. USAID should also seek support from other institutions that work in health so that actions still in process when at the program end are carried through. The Itinerant Brigades The strategy of using IBs to provide comprehensive health services to excluded and dispersed populations has been successful, even though sustainability of the strategy will be an ongoing challenge due to the inherent cost of serving scattered populations living far from the nearest health center and without adequate roads and river transportation. The IBs also confront cultural beliefs associated with immunizations that result in suspicion and at times their rejection making it important for the IB team to deal with issues at the local community level. There are opposing opinions within the Peruvian health system on the efficacy of using IBs to work with excluded and dispersed populations: some consider that the budget assigned to the IBs could be used to hire health center personnel to perform the same duties, making the intervention more sustainable; or that primary health facilities could be built up with more staff, better equipment and infrastructure, and increased educational capacity, with the goal of making the IBs unnecessary. The sustainability of the strategy to provide the IBs with adequate sustainable funding is dependent on political interest in their operations, both nationally and regionally. The governing institution, the MOH, and the regional governments should recognize their importance and understand that it is through them that the excluded and dispersed populations have access to comprehensive health services. As part of decentralization, the GOP commitment will be a consequence of awareness created among political authorities nationally, locally, and regionally, to grant the IBs an increasing role in the development of healthy communities. This would require working closely with other institutions such as the National Registration of Identification and Marital Status (RENIEC, Registro Nacional de Identificación y Estado Civil) or strengthening relationships with other MOH departments, such as sanitation, environment, and nutrition. If this does not happen, even though IBs are incorporated in the regional budget, they will not be given the importance they merit strategically, and the budget could be redirected to other 48 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT activities. With a powerful awareness campaign, the number of IBs could be increased in the regions having large vulnerable populations and more health centers could rely on their help. The Immunization Information System The main objective in strengthening the current MOH immunization information system is to more efficiently report on children immunized with the complete course of vaccines, though it would also be useful for MOH forecasting, planning, budgeting, and logistics. The Threshold Program is providing support to implement the IIIS (incorporated in the SIS), which facilitates follow-up of immunized children via a detailed registry by name and generates detailed reports, preventing both lost opportunities and duplicate vaccinations. The sustainability of an initiative of this magnitude is not assured by a two-year program—it takes much more time. However, there are actions and strategies that could lead to sustainability. The most important is approval of the IIIS Directive that would go a long way to guarantee sustainability and ensure expansion to the entire country. The HIS is the overall Health Information System. SIS is a component part of that system covering comprehensive health insurance. Approval of the directive would make the IIIS an official part of the MOH information system. In its latest statement (memorandum No. 685-2010-OGEI/MOH) the OGEI indicates it has no objections to the proposal of Directive IIIS; and the SIS (document No. 870- 2010-SIS/J) has made observations that should be included in the proposal for the IIIS directive that should be put forward for approval. Along with recognition of the new information system via approval of the IIIS Directive it will be important to update the regulations governing the health sector information system to cover the new system. Clarifying the relationships between areas that are involved with production, collection, and analysis of information is important: The General Statistics and Information Office, National Immunization Health Strategy, and SIS., As is well known, the rules that define the organization of the MOH (D.S Nº 023-2005-SA published in January 2006 and modified by D.S Nº 007-2006-SA of May 2006, MOF) establish that systems for production, analysis and publication of information by the MOH are the responsibility of the General Statistics and Information Office, ―the organ responsible for the production, analysis and publication of statistics in the Health Sector‖ (article 28), and the General Directorate of Epidemiology, which is ―responsible for conducting the Epidemiology Supervision System in Public Health and the analysis of health in Peru.‖ This same law mentions that the General Directorate of People’s Health, which includes the National Health Strategies, is not assigned any function related to information systems. Generating improvements in an MOH information system without a directive and permanent coordination between the offices handling information—OGEI, ESNI and SIS— risks having their use restricted if it is not considered ―official,‖ For example, in Amazonas, an internal memorandum emphasizes that the HIS Information System is the only information source within DIRESA and urges that it be strictly implemented, supervised, and monitored. Also, besides the legislation, efforts should be continued to reinforce the skills of health personnel for drawing up plans for supervision, analysis of information, and immediate decision-making on the basis of information results, as well as on the use of IIIS. Another important sustainability factor, which is being resolved, is the need for SIS to rely on equipment (a server) that would allow for a registry to improve the quality and rapid access to vaccination information before IIIS is connected nationwide. During the assessment review meeting discussion, USAID indicated that project funds might be available for the server. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 49 Management and Logistics Systems The fact that new logistics software has not yet been created jeopardizes the sustainability of the management and logistics systems for basic children’s immunization. Instead, the GOP is strengthening its existing administrative system; SIGA is used in every State institution. Using SIGA will mean that the immunization system can rely on a regional and national map of the cold chain, which will facilitate evaluation of the status of the cold chain from different levels of service, allowing decisions to be made and action taken almost immediately to solve problems. However, the catalog of equipment acquired before 2007 is not complete, and its approval by MEF and its registry in the software is still pending. It is important that work continue and actions taken to complete this activity. Likewise, the training of regional personnel in the use of SIGA software needs technical support from the MEF. Another requirement for sustainability is that the DIRESAs have budget and funding for maintaining the cold chain and that inputs and human resources to maintain and repair the equipment are available to the networks/health centers. Budgeting and resource management in the DIRESAs are in transition as the GOP implements its decentralization strategy. Not all DIRESA administrators are equally clear on their new responsibilities. Among the strengths: some of the regions have acquired new equipment (for the cold chain) to be distributed according to the requirements of the networks. This helps guarantee the capacity for storage at the DIRESA level. Initiatives for Continuation of Threshold Program Activities In this final stage, the Threshold Program implementers should create space for discussion with their GOP counterparts, examining issues and problems encountered so far, and facilitate discussions on alternative strategies for moving forward using the recommendations found in this report. The counterparts themselves should also propose actions they can commit to implementing. Threshold Program implementers might also consider sharing the findings and recommendations with counterparts and other institutions engaged in health services and immunizations, including the World Bank, the IDB, and the PAHO. COORDINATION BETWEEN THE MOH AND REGIONAL GOVERNMENTS As long as the people in political and management positions, at executive and assistant levels consider the contributions of the Threshold Program to be important, sustainability will be achieved in spite of the approaching changes in local, regional, and central government. This is why it is important to create awareness among national and regional authorities as well as health personnel of the results the program has achieved for the dispersed populations and the interventions that made it happen. It is therefore fundamental to continue strengthening the National Immunization Health Strategy, regionally and nationally, so that the players consolidate as a committed team with the skills required to continue actions already initiated and deal with new challenges within the decentralization process. Findings, Conclusions, and Recommendations ITINERANT BRIGADES Findings: The joint effort of the MOH and the Threshold Program to increase coverage of children’s basic immunization in rural areas using IBs to serve rural populations has produced the following results: 50 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT The NTS No 081-MOH/DGSP V.01 Technical Health Standard for the comprehensive health service teams to excluded and scattered populations was updated (approved July 21, 2009). DU No 076-2009 sets out extraordinary measures for the execution of the IB programs that were operating in 2008 and stopped because of a lack of funds; it authorizes the MOH to transfer budgetary credits to finance the operation of IBs in 16 GR (regions) for up to S/.9,305,407. In November, via DS No 243-2009-EF, a transfer (for fiscal 2009) of S/.8,216,457 was authorized to implement and operate the teams in Amazonas, Ancash, Apurimac, Ayacucho, Cajamarca, Cusco, Huancavelica, Huanuco, Junin, La Libertad, Loreto, Madre de Dios, Pasco, Puno, San Martin, and Ucayali. The DGSP and the OGEI together drew up and assigned codes to the 17 DIRESAs that are equipped with this kind of mobile health unit, which will help unify the registry of services they provide (Document No 1254-2010-DGSP/MOH), allowing future registration of the specific interventions carried out by the IBs. It will also allow for reimbursement for SIS assistance and permit its incorporation into the SIGA PPR, which the MEF is currently processing. Equipment has been procured for 151 IBs, financed by the Threshold Program (at the request of the MOH [2009 Document]), and has been distributed in 17 regions. Training for coaches, in five modules, was prepared jointly with the MOH Health team (which guarantees its use once the program ends), with technical and financial support from the Threshold Program; 163 people were trained in Phase 1 and 786 in Phase II. The Threshold Program is providing TA to ensure regional funds for 2011. The DIRESA Executive Units have designed their budgets to cover the work of the IBs as part of budgeting for results (BFR). Based on this, the MOH is drawing up a joint budget that will be presented to MEF for approval and which will ensure the IB budget for 2011. A workshop in May 2009, ―Strengthening the Skills of IB Regional Coordinators for Programming Activities and Budget Management to Serve IB Populations,‖ trained personnel on how to draw up the budget and transfer it to PPR–SIGA software. Participants were IB regional coordinators from the 17 regions. PDA equipment has been acquired to facilitate sending of data collected by the health centers. Software is being designed to be incorporated into this equipment. The skills of the IB regional coordinators were reinforced in terms of programming activities and budget management for serving excluded and scattered populations in terms of the MEF BFR. The program seeks to increase demand from local and regional authorities and the public with material printed in five languages (Spanish, Quechua, Aymara, Awaruna, and Ashaninka) and audio material in these languages plus Shipibo and Huambis. The materials were developed in alliance with the MOH Directorate of Health Promotion and General Communications Office. The program is also providing support in the training of community agents in immunizations and promotion of attitudinal change among schoolchildren with the Cuido a mi Familia Plan (I take care of my family) promotion, in accordance with the recommendations of the Survey of Knowledge, Attitudes, and Practices conducted by the program. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 51 There are still, however, limitations to the strengthening process: a) At the time of the assessment, four IBs of the Ayacucho DIRESA no longer function, leaving nine still operating, The 2010 budget assigned by MEF for Ayacucho took as reference the immediate past history of the IB, i.e. nine teams, and the regional government did not support an increase to 13. Of the nine active mobile teams, some do not have the full five members: physician, nurse, obstetrician, dentist, and health technician because of the rotation of personnel and forthcoming appointments in the city. This emphasizes the importance of follow-up and periodic reports about team operations. b) In some regions, acquired equipment has not been delivered to the IBs. In some regions, the regional government’s participation in defining the budget has been limited, so personnel linked to the IB programs are unclear about what will happen with budgets in 2011. This is true in DIRESA- Ayacucho, which is an example of what could happen in the future if the use of IB resources is not made clear. The Threshold Program has provided TA for this process, despite frequent changes of counterpart personnel and a lack of clarity about the roles of authorities in the decentralization process. Conclusions The IBs have a budget for 2010 and a budget planned for 2011. They have been assigned codes, and equipment has been purchased for them through the Program; however, over the long term coordination between DIRESA, OGEI, and ESNI-DGSP must be closer. The program updated the Technical Health Standard of the Comprehensive Health Service Equipment for Excluded and Scattered Populations; codes were assigned for reimbursing services by the teams, which will allow for future registration of specific interventions carried out by the IBs; the program trained coaches for IBs; and equipment was acquired for 151 teams (at the request of MOH). Equipment has been distributed among the 17 regions, although not all has reached the teams (as in the case of Ayacucho). In the 2010 budget the MEF transferred funds to the regions, but for 2011, related to decentralization and BFR, the regions have had to draw up their own budgets. The Executor Units have done so, with Threshold Program help. The MOH is drawing on this information for its own budget, which will be presented to the MEF for approval, guaranteeing the budget for 2011. Because the sustainability of the IBs depends on what the regional governments can accomplish, it will be important to plan and take action during the time left in the program to create awareness, set priorities, and built political will. It is noteworthy that in Ayacucho, Amazonas, and Ica, staffs of the regional governments and others did not clearly understand how the IBs would be financed in future. Among options mentioned were universal coverage, investment projects, participatory budgets, and BFR. A factor that contributes significantly to the sustainability of the IBs is that some regional governments have incorporated the IBs directly into their budgets, as have Cajamarca, Cusco, Loreto, Puno, San Martin and Ucayali. Currently, 16 teams operate this way. Skills are being built so that it will be possible to rely on a regional nucleus capable of carrying out cascade training for new IB staff. The IBs have greater stability as part of a micro network because their information is incorporated into the center to which they are assigned, or they function as an Executor Unit. In the past, ―production‖ information was handed in with payment reports, and information was lost. 52 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Recommendations Revise the technical standards so that the IBs can be regulated according to guidelines (to be drafted) that can help their members understand what to do in exceptional cases. , Share the experience with all institutions that include flexible intervention in their lines of work. Ensure that the service provided by the teams is of high quality, supported by a dependable budget, appropriate equipment, and trained staff, as the program provides. To achieve the desired results, training is essential. With the turnover in the IBs, the efforts invested can easily be lost, so a permanent training plan for new IB members should be drawn up in each region. The Threshold Program has trained 163 coaches in the 17 regions who will be responsible for providing continuity to the training. Besides printed training material to be given to the regions, low-cost materials could be made available through Web access or on CDs. In order to follow up on the actions performed by the IBs, based on Annexes 6 and 7 of the Technical Standards, define more clearly what is to be supervised and monitored, particularly in the case of immunizations and their relationship to the Flexible Offer. Even though equipment is already included as part of the assets of each DIRESA, draft maintenance and replacement plan before the program ends. The IB program is hard on equipment, making it doubly important to take replacement into consideration. Future Actions Involve MEF and regional government planning teams, with representatives of all departments that would be directly or indirectly involved in the work of the IBs, in the support, programming and execution of the budget for the teams. Also undertake advocacy activities with the health system to reinforce the importance of the teams for serving deprived populations. Nationally, strengthen the MOH Directorate of Services so that it can take an even stronger position in its leading role related to the Flexible Offer. This would allow it to update its training, equipment, and budget plans. Involve other institutions and NGOs directly or indirectly concerned with immunization in monitoring the IB teams. External follow-up and control could be implemented from the community, which would evaluate and qualify IB actions in addition to the control exercised by the State, which is generally limited to delivery of information from the teams and to annual supervision. MANAGEMENT AND LOGISTICS SYSTEMS FOR IMMUNIZATIONS Findings Another of the Threshold Program’s objectives is to build up logistics and management systems for basic children’s immunizations. The program, through ESNI, has been encouraging the cataloguing of equipment and the standardization of tools to incorporate the cold chain inventory into the SIGA asset registry. Among accomplishments are: Equipment acquired after 2007 has been coded and the catalogs approved by the MEF and incorporated into SIGA registers. Meanwhile, the regions have been incorporating into SIGA assets equipment they acquired before 2007. Before 2007, each region or network PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 53 had its own data base, which made it impossible for management to comprehend the status of regional and nationwide cold chain and greatly complicated decision-making. Because the program is nearing an end, training in SIGA software will be carried out with the information already incorporated on equipment acquired since 2007. A catalog being developed to incorporate the old equipment into the SIGA asset register must be approved by the MEF before it is updated in the software. Virtual Specialization Course in Management of Immunization tools are being provided that will permit regional teams to prepare their own budgets, taking into account their differing situations, such as operations costs, transportation, and equipment maintenance. Another aspect of the immunization management and logistics system is related to requisitions. The requisition for immunization inputs required by the MEF, using PAHO’s Format 173-1S, does not guarantee timely delivery of vaccines. Although vaccines for 2010 were requested there have been delays in delivery of those for polio and yellow fever. This contributes to a drop in coverage, even though recovery can be made with vaccination campaigns. In the case of MMR (immunization shot against Measles, Mumps, and Rubella), there is still a time lag in delivery, although this has not resulted in a shortage because there is a reserve stock. There are also still some limitations: There is a delay by MEF in preparing, updating, and approving the catalogs, resulting in a delay in updating the inventory software for the computers acquired before 2007, information for which had been collected only numerically. This means that these computers cannot be incorporated into the SIGA assets. The requisition of immunization inputs by the MEF, via Format 173, does not guarantee timely delivery of the vaccines. The process begins in the first semester (global demand) so that by July the budget may be allocated for the following year. An international bid can then begin. The price list is issued in February of the following year. This process causes delays from the time of acquisition to the time of delivery of the immunization supplies. Conclusions As of 2007, computers have been incorporated into the SIGA asset registry; the cataloguing and incorporation of older computers is pending and will require a significant effort by the implementing team and the MOH. Incorporation of the cold chain in the SIGA asset registry will mean that a regional map of it will be available for local as well as regional administration, micro, and decision-making networks. The availability of vaccines depends not only on MEF requisitioning but also on programming, budget formulation, delivery, storage, and distribution. Recommendations Follow up with the DIRESAs to continue incorporating older equipment into the SIGA asset registry. Once the old equipment is cataloged and the catalog approved, it should also be incorporated. Centers with equipment that must be inventoried and incorporated into SIGA need to be accessible, so that the information can be collected from all centers nationwide, as a medium-term achievement. 54 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Training sessions related to the cold chain should be addressed to personnel not only in the Immunization Health Strategy but also in the office of the area managing the budget for the cold chain. The Planning Office and other relevant areas of the MOH and DIRESA should also be involved as well as the MEF teams. Provide TA to the DIGEMID/DARES teams jointly with the MOH planning and budget teams and the ESNI itself to firm up the process of vaccine programming, requisitioning, acquisition, and distribution. Also, promote coordination between these teams and the team on immunization health strategies. Future actions Since Peru has an expansive vaccination scheme, it is necessary to consider the infrastructure that sustains it. Coordination and advocacy should take place with regional and local governments so that they include health activities in their budgets, such as maintaining the cold chain and building vaccine storage capacity in regions and centers. Close the gap between the third level that has or will have refrigeration and the first and second levels, which have storage difficulties, by promoting small investment projects for cold chains, locally and regionally. Strengthen the process of planning, programming, requisitions, acquisition, and distribution of vaccination inputs, working closely with the National Immunization Health Strategy, the General Planning and Budget Office, the MEF, and the PAHO. INTEGRATED INFORMATION SYSTEM ON IMMUNIZATIONS Findings The MOH is implementing the IIIS with the technical and financial support of the Threshold Program in order to establish an effective, timely, and appropriate nationwide information system, facilitating the development and evaluation of ESNI interventions by providing a single reliable source of information on immunization coverage of children under 5. It is using the platform of the SIS information system, which allows follow-up of immunized children by name. The system is being implemented by scaling up at the regional level and gradually incorporating additional centers. To do this, through DIRESA Directorate Resolutions, IIIS Implementation Committees of the Health Regions have been set up and are responsible for complying with the activities and regulations for application operational integrity. DIRESA reports on progress are transmitted by letters or official documents. The positive results of this intervention are the following: Gradual regional scaling up of the IIIS; DIRESAs, nationwide, have IIIS. Facilitation of follow-up of immunized children by name. Guarantees of vaccine for all children, preventing lost vaccination opportunities where it is not known where a child to be immunized is from. Previously the child might be redirected to its centers of origin to be immunized. Now, the system reports the child as being registered (or not) at that center (names, surnames, codes), allowing cross-checks of information and providing certainty for follow-ups. Vaccine is not ―lost‖ because duplication is avoided. This is an issue especially for migrant children. In the management agreements of some regions, the total number of children has been counted according to the registry of names. PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 55 So far, 1,246 health centers have received computer equipment (agreement between Abt and USAID). This equipment is being used by immunization strategy coordinators to facilitate follow-up of coverage. Although IIIS coaches have not been trained, training has been given to regional teams of implementers that include the immunizations strategy coordinator, who is responsible for the Statistics and Informatics Office, and the person responsible for ODSIS in operational processes and SIS regulations. This has facilitated use and sharing of IIIS information. There are, however, still some limitations: Without a directive, mechanisms for coordination between the OGEI and the SIS and the SIS and DIRESAs have yet to be established, which means that the SIS does not officially know how many centers have been incorporated into the IIIS and, the networks and centers do not receive feedback from the SIS. There is a reporting delay of about one month. The consolidations issued by the Health Coverage System are referred to centers and regions and a nominal consolidation allows for an inspection of the last vaccine the child received. It does not issue a nominal consolidation showing the details of each vaccine a child has received, and so each center works with its own ―Excel nominal consolidation database‖ to follow up , making double work for the staff. The HIS reports a greater number of vaccines than the IIIS. Generally, this is the result of inadequate filling of the SIS Single Service Form. Regarding the use of codes and reports: (i) There is no standardization in the use of CIE (International Nursing Council) 10 codes between the HIS (z276) and the SIS (z278), which can cause confusion for staff. (ii) Although the SPR is only one dose, in the IIIS report it appears as two doses. For reports, coordinators assume what it is, a single vaccine. The training of coaches is still pending because approval of the directive is pending. Conclusions The IIIS pilot has been successfully implemented, initiating nationwide scaling up, which facilitates ESNI’s task of following up vaccinated children through the nominal registry and no longer losing vaccination opportunities. Approval by the directive is still needed, to consolidate it nationwide. The IIIS is being expanded nationwide through gradual scaling up in regional health centers; and its success depends to a great degree on coordination between the DIRESAs, ESNI (through ESRI), the SIS (through ODSIS), and the OGEI (through the OEI). Approval of the IIIS directive is vital for institutionalization, since that will allow for official implementation nationwide and ensure that scaling up is not halted. Even though this system is being implemented, and the centers can have access to the results, feedback must be secure, since the centers do not receive any formal feedback from the SIS on information collected. The ODSIS has not been formally informed of the IIIS expansion in the DIRESAs (number of centers that have begun scaling up). IIIS has had limited acceptance in the Statistics and Informatics Offices, both regionally and nationally. 56 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT Recommendations In order to implement, expand, and consolidate IIIS nationwide: Follow up intensively up on approval of the IIIS directive and then publicize it widely. Implement strategies to reinforce those in institutions and departments (ESNI, OGEI, SIS) who are directly or indirectly involved with IIIS and its implementation. The issue is its importance in the nominal registry of immunized children and their follow-up. Continue holding technical and training meetings to help people use IIIS appropriately and discover its efficiency in following up children’s immunizations. Encourage working in synergy and coordination and incorporate the ESI, ODSIS, and OEI teams in the training sessions to be held so that they become involved in the system and share the team vision rather than each defining its role in the system independently in a vacuum. Future Actions Advocate at the senior management level (MOH and MEF) so that if additional equipment is needed for collection, analysis, and dissemination of data, it can be incorporated into the budget under the heading of universal insurance. Advocate with other institutions, NGOs, and projects for strengthening information systems so they take up the challenge of continually reinforcing the IIIS. This is especially important in the context of recent and coming political changes. General Recommendations for Immunizations a) In the context of decentralization, continue to support the members of the Immunization Health Strategy and the teams taking part in them: Management Services, Planning and Budget Office, Statistics and Information Office, Comprehensive Health Insurance, DIGEMID/DARES. National and regional levels require encouragement so they will be a confident, committed team able to continue actions already underway and deal with new challenges. b) Give IB a complementary role in building healthy communities, working with other institutions like RENIEC and state programs like JUNTOS, and strengthening relations with other MOH units such as environmental sanitation, and nutrition so that implementation not only promotes health but also influences the quality of community life generally. c) Strengthen the use of IIIS nationally by validating relationship between areas involved in the production, collection, feedback, and analysis of information. This could begin with workshops or technical meetings to promote coordination with the ESNI, INEI, and SIS nationally and regionally. d) Firm up the process of planning, programming, acquisition, and distribution of vaccine supplies. Build health staff capacity sector and disseminate the progress of implementing BFR (USAID could adapt material from other USAID projects). PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT 57 APPENDIXES The following appendixes are published in Volume II. Appendix 1. Results Reporting Table (RRT) Appendix 2A. Evaluation of Indicators and Milestones (English Version) Appendix 2B. Evaluation of Indicators and Milestones (Spanish Version) Appendix 3. Outline of Structured Interviews with Counterparts and Implementers for Anti-Corruption Component (Spanish Version) Appendix 4. Outline of Structured Interviews for the Immunization Component (Spanish Version) Appendix 5. Outline for Focus Groups (Spanish Version) Appendix 6. Calendar of Activities of the Consultancy (Spanish Version) Appendix 7. List of People Interviewed (Spanish Version) Appendix 8. Data Flow Diagrams(Spanish Version) Appendix 9. Bibliography(Spanish Version) Appendix 10. Scope of Work (English Version) Appendix 11. CONECTA Document (Spanish Version) Appendix 12. CONECTA Survey Instrument (Spanish Version) 58 PERU THRESHOLD PROGRAM (PROGRAMA UMBRAL PERU): DATA QUALITY REVIEW AND MIDTERM ASSESSMENT For more information, please visit http://www.camris.com CAMRIS International 6931 Arlington Road, Suite 575 Bethesda, Maryland 20814 Tel: (301) 770-6000 Fax: (301) 770-6030 www.camris.com