Avoiding Data Graveyards: Insights from Data Producers & Consumers

Editors: Samantha Custer and Tanya Sethi

Contributors: Deirdre Appel, Carmen Cañas, Jose Sierra David Castillo, Taryn Davis, Harsh Desai, Dina Abdel-Fattah, Lauren Harrison, Paige Kirby, Oussenyou Ngom, Caleb Rudow, Vanessa Sanchez Jacob Sims, and Alena Stern

Acknowledgments

This research was conducted under the direction of Samantha Custer (AidData, College of William & Mary). This publication would not have been possible without the contributions of an exceptional group of authors that lent their time and talents to write the three country studies that form an important foundation for this research. These individuals include: Harsh Desai, Lauren Harrison, Tanya Sethi, Jacob Sims, and Alena Stern (AidData, College of William & Mary); Dina Abdel-Fattah, Carmen Cañas, Taryn Davis, Paige Kirby, Ousseynou Ngom, and Vanessa Sanchez (Development Gateway); and Jose Sierra David Castillo, Caleb Rudow, and Deirdre Appel (University of Texas-Austin). John Custer and Sarina Patterson (College of William & Mary) helped with the layout and copy editing of this publication.

We owe a debt of gratitude to those who participated in key informant interviews and answered our questions, as their insights offer an invaluable perspective on the constraints and opportunities for development data to shape decision-making. Vanessa Goas and Josh Powell (Development Gateway) and Dr. Catherine Weaver (University of Texas-Austin) were invaluable in consulting on the research design and the drafting of this report.

The authors appreciate the wonderful group of peer reviewers that helped refine our thinking and prose. These individuals include: Dr. Ariel BenYishay, Agustina Eskenazi, Carey Glenn, Dr. Brad Parks, David Trichler, and Alex Wooley (AidData, College of William & Mary); Nancy McGuire Choi (Development Gateway); Johannes Tonn (Global Integrity); Stefaan Verhulst (GovLab, New York University); Laia Grino (InterAction); Michael Jarvis and Lauren Keevil (Transparency and Accountability Initiative); and Kimberly Smith (United States Agency for International Development). We also thank representatives from the USAID missions in Honduras, Senegal, and Timor-Leste for their feedback and assistance in conducting this field research.

This research was conducted with generous support from the United States Agency for International Development (USAID) Global Development Lab (through cooperative agreement AID-OAA-A-12-00096) and the William and Flora Hewlett Foundation. The views expressed here do not necessarily reflect the views of USAID, the United States Government, or the William and Flora Hewlett Foundation. This study also supports AidData’s commitments as an anchor partner of the Global Partnership for Sustainable Development Data (data4sdgs.org) and an institutional member of the Global Open Data for Agriculture and Nutrition (http://www.godan.info/) initiative. It was jointly conducted by an interdisciplinary team of faculty and staff based at the College of William & Mary, Brigham Young University, Development Gateway, and the University of Texas-Austin.

Citation:

Custer, S. and T. Sethi (Eds.) (2017) Avoiding Data Graveyards: Insights from Data Producers and Users in Three Countries. Williamsburg, VA: AidData at the College of William & Mary.

List of Acronyms

ACDP: AidData Center for Development Policy

AIMS: Aid Information Management System

AIP: Annual Investment Plan

AMP: Aid Management Platform

ANSD: National Agency of Statistics and Demography (Senegal)

ATP: Aid Transparency Portal (Timor-Leste)

CENISS: National Social Sector Information Center (Honduras)

CSOs: Civil Society Organizations

DCEF: Division of Economic and Financial Cooperation (Senegal)

DFAT: Department of Foreign Affairs and Trade (Australia)

DPMU: Development Partnership Management Unit (Timor-Leste)

DPRE: Directorate of Planning and Education (Senegal)

EITI: Extractives Industry Transparency Initiative

ESP: Emerging Senegal Plan

FOI: Freedom of Information

GoH: Government of Honduras

GoS: Government of Senegal

GoTL: Government of Timor-Leste

GPSDD: Global Partnership for Sustainable Development Data

HDI: Human Development Index

HMIS: Health Management Information System (Timor-Leste)

IDB: Inter-American Development Bank

IMF: International Monetary Fund

IPs: Implementing Partners

INE: National Statistics Institute (Honduras)

INEADE: National Institute of Research and Action for the Development of Education (Senegal)

LMs: Line Ministries

MAF: Ministry of Agriculture and Fisheries (Timor-Leste)

MCC: Millennium Challenge Corporation

MDGs: Millennium Development Goals

M&E: Monitoring & Evaluation

MERECE: International Cooperating Partners’ Education Round Table (Honduras)

MoE: Ministry of Education

MoF: Ministry of Finance

OECD: Organization for Economic Co-Operation and Development

ODA: Official Development Assistance

PGC: Plataforma de Gestión de la Cooperación (Honduras)

SACE: Sistema de Administración de Centros Educativos (Honduras)

SAMI: Sistema de Administración Municipal Integrado (Honduras)

SCGG: Secretariat of General Coordination (Honduras)

SDGs: Sustainable Development Goals

SDP: Strategic Development Plan

SEFIN: Secretariat of Finance (Honduras)

SGPR: Sistema de Gestión por Resultados (Honduras)

SIAFI: Sistema de Administración Financiera Integrada (Honduras)

SLMS: Suco-Level Monitoring System (Timor-Leste)

SRECI: Secretariat of External Relations (Honduras)

UN: United Nations

UPEG: Planning and Management Evaluation Unit (Honduras)

Chapter 1

Avoiding Data Graveyards: Insights from Data Producers & Users

Authors:

Samantha Custer and Tanya Sethi

Amina Mohammed recounted seeing sacks of paper — data points on Nigeria’s development progress or lack thereof — left forlornly on the floor with little chance of being seen or used. Ms. Mohammed, speaking at a 2014 workshop on the data revolution for sustainable development,2 explained that the experience helped shape her vision as the United Nations Special Adviser on Post-2015 Development Planning (Post 2015 Data Test, 2014).3 A revolution that stops at producing more and better data will ultimately fall short of the aspiration to improve people’s lives.

Ms. Mohammed’s experience is not unique to low- and middle-income countries or to paper-based information. The human costs of data disuse in advanced economies hit the headlines in 2007, when New York Times reporters exposed chronic mistakes and delays in care for wounded soldiers as medical personnel in military hospitals failed to use existing data management systems for patient care (Urbina and Nixon, 2007). The article attributed this failure of care to the inability of the United States Department of Defense to effectively mainstream new norms of decision-making based upon data rather than personal discretion.

Over the past decade, there has been a seismic shift in the world’s capacity to produce data to monitor progress and inform action on sustainable development (United Nations, 2014). Nonetheless, the power of data to change development outcomes ultimately rests on its ability to inform policymakers as they allocate resources, evaluate results, and make course corrections. In other words, the data revolution could succeed in building a supply of better development data, but may falter if there is insufficient demand for its use. The two earlier anecdotes are a cautionary tale: a ready supply of development data that lies fallow from disuse is little more than a graveyard, a place where data goes to die.

Why are we seeing this disconnect between the supply and demand for development data? In short, we have a stunted feedback loop problem – data producers are often far removed from the people they hope will use this information to make decisions and advocate for reforms (Custer et al., 2016; Development Gateway, 2016). Moreover, our capacity to produce data is rapidly outstripping our ability to understand who uses this data, in which contexts, and why.

In this report, we attempt to remedy this evidence gap by shedding light on the demand for data by development decision-makers and those that advise them in three countries – Honduras, Timor-Leste, and Senegal. We explore the barriers to the supply of, and demand for, development data, as well as identify opportunities for greater alignment of available data with the information that people want to support their decision-making in specific contexts. Drawing from the three country studies, we distill a set of broader findings and recommendations to inform future investments in production and use of sustainable development data.

Section 1.1
The Status Quo:
Market Failures in Development Data

Once the provenance of governments and multilateral institutions, a broader constellation of private, public, and civil society actors are producing more varied data points on development efforts at greater quantities than ever before (Parks et al., 2015; Custer et al., 2015). New technologies and methods have dramatically reduced the cost, time, and effort to collect and publish data with greater frequency and higher precision (United Nations, 2014).

Governments and development partners are increasingly opening up their data on investments and results. Line ministries, national statistics offices, and parliamentarians are also responsible for creating, disseminating, and using data. Citizens, civil society organizations (CSOs), and companies are no longer merely consumers of information, but are creating their own data points on the quality of public service delivery and the health of their communities.

Who are the target audiences for this data? Producers and funders of development data often have a rough idea of the key constituencies they are trying to reach. Researchers and evaluators need quality assured, raw statistical data to assess the impact of development efforts over time. Citizens and policymakers want real-time qualitative and quantitative information to influence key decisions now. Governments and development partners are looking for data disaggregated by geography (e.g., national and subnational) and demography (e.g., gender, age, disability status) to ensure that no one is left behind. A common need across these groups is data that illuminates connections between resources, results, and outcomes.

Yet when it comes to aligning supply with demand for development data, the devil may be in the details. Producers and funders of development data are often limited to constructing vague, and arguably naïve, archetypes of their ideal users. Read and Atinc (2016) adroitly capture two of these common archetypes as: (1) the superbureaucrat who has the time, ability, and incentive to make evidence-informed decisions and (2) the supercitizen who uses data to hold service providers accountable for service delivery.4 One might also add a third generic archetype to this list, the superjournalist, who has a ready grasp of data and statistics to weave into their news articles to inform and provoke action for sustainable development.

In this idealized view, opening up information on development inputs (money) and outcomes (results) “catalyzes a chain reaction that mobilizes citizens, politicians, and frontline service providers” to use resources more effectively to improve people’s lives and livelihoods (Custer et al., 2016). However, little is known about how these user groups truly respond to existing development data, whether they find this information to be helpful in their work, and the circumstances under which they deem it worth the effort to put data to use.

In this study, we assess the enabling environment for data production and use in the context of three countries, applying a theory of change (ToC) previously developed by Custer et al. (2016).5 This ToC presents the causal logic of getting from data to impact as the interaction of four C’s: content, channel, choice, and consequence. Governments and organizations disclose data on development resources and results (content), disseminate this information to users online or offline (channel), whereby citizens and officials take action individually or collectively (choice), to improve the country’s performance on achieving sustainable development for all (consequences).

Funders and producers make several assumptions as they design, collect, and disseminate development data – from what they choose to measure and the quality of the data collected to the modalities they use to transmit this information to their intended audiences and the incentives that exist for people to act upon it (Custer et al., 2016). These assumptions will either prove to be “correct or fundamentally flawed” and data producers have substantially less control as they move down the results chain from inputs to outcomes (Custer et al., 2016). When these assumptions hold, they can serve as enablers for greater use of development data. However, when they break down, there is a higher risk of producing data graveyards, instead of data for action.

Absent a visible demand for development data from citizens and officials, there is little incentive for those funding or producing data and statistics to change the status quo (Custer et al., 2016; Development Gateway, 2016). Additionally, when existing development data is not seen as sufficiently timely, relevant, and credible for their purposes, it is unlikely that there will be an uptick in demand by citizens and officials for this information.

Section 1.2
The Desired Future:
Catalytic Investments in Data for Action & Monitoring

The consequences of producing or publishing data that citizens and decision-makers do not put to use are not trivial. Domestic and international stakeholders expend substantial resources to collect and publish development data. Yet even these current investment levels are likely not enough to meet the challenge of the post-2015 era. Resources for data and statistics are particularly scarce in low- and middle-income countries. For example, a report on the State of Development Data Funding (Badiee et al., 2016) estimates a global financing gap of $350-400 million per year to produce statistics to support monitoring of progress against the Sustainable Development Goals (SDGs). 6

Collecting data imposes a reporting burden on overstretched officials, requires effort to derive meaningful insights, and diverts resources from other important activities (Jerven, 2014; The Economist, 2015; Development Gateway, 2016). Concurrently, the persistent absence of revealed demand undercuts the efforts of open data champions to push for greater transparency or convince data holders of the value of investing time, money, and attention to share and disclose information. This unused data yields a low return on investment.

Nonetheless, building a supply of better development data could be a catalytic investment if the chronic mismatch between demand and supply was addressed (Custer, 2014; Davies, 2014). This implies the need for a closer marriage of producers and users of data along the entire continuum of collecting, aggregating, publishing, and interpreting data (Khan and Foti, 2015). It requires identifying entry points where information can make a difference (Custer et al., 2016). The stages of production and use should be symbiotic and mutually reinforcing, rather than distinct considerations where producers worry about use at some future point.

The data revolution poses a galvanizing challenge to governments, development partners, and civil society organizations (CSOs) to bolster both the supply and use of development data for planning, monitoring, and reporting (Badiee et al., 2016). In September 2015, the Global Partnership for Sustainable Development Data (GPSDD) formed to address a fundamental problem: “unreliable or non-existent data and the lack of skills and willingness to use it” that hamstrings efforts to achieve global goals and national development priorities (GPSDD, 2016).7

Fixing this market failure requires closer attention to what end users want from their data – what are the most pressing gaps, in what cases will data be used, and how will it be financed sustainably? As a long-standing producer of data on financing for sustainable development, answering these questions is of both academic interest and practical importance to AidData and our coalition of partners under the AidData Center for Development Policy.8

Section 1.3
This Study:
Approach, Methodology, and Value Add

There is a growing body of literature on topics related to data use and the impact of transparency and accountability initiatives;9 however, these studies seldom assess the political economy of how data is produced and used to support specific types of decisions. A unique value add of this report to the broader literature is that we situate the discussion of enablers and constraints to data use in the context of how government, development partners, and civil society actors target, monitor, and evaluate the impact of development finance in their countries.

In particular, this report endeavors to answer the following research questions:

Between October 2015 and September 2016, the research team conducted a desk review of relevant literature and key informant interviews in three countries to better understand the enabling environment for the use of development data — including, but not limited to that produced by AidData — in decision-making. In designing the study, we sought to capture both a country-level view of how data on development investments and results is being used in specific decision-making processes, as well as deeper-dive assessments in three focus sectors: education, agriculture, and health.

In determining our population of interest, we sought to emphasize those actors most likely to be in a position to inform or make decisions regarding the financing, allocation, or targeting of development activities. The research team, therefore, focused its efforts on three core user groups that fit this profile: government officials in central and line ministries, development partner representatives based in country, and leaders of prominent civil society organizations. We recognize that citizens, companies, and parliamentarians are important potential consumers (and producers) of development data; however, we determined that these user groups were ultimately outside of the scope of this particular study.

The research team selected Honduras, Timor-Leste, and Senegal as the three countries for the study on the basis of several criteria. The first winnowing criteria for our country selection process was the desire that this study provide a feedback loop to the AidData Center for Development Policy’s (ACDP) own work on geocoding aid information funded under the auspices of the USAID Global Development Lab’s Higher Education Solutions Network program.10 This restricted the pool of potential countries to nine candidates: the Democratic Republic of the Congo, Haiti, Honduras, Nepal, Niger, the Philippines, Senegal, Timor-Leste, and Uganda. In further refining this list, we sought to select countries that varied on the basis of region, income, open data commitments, and Internet connectivity.11 We also considered the quality of available geocoded aid data in the country as a precursor to assessing use of this information.12

In each country, we analyzed data use in two sectors. In choosing the focus sectors in each country, we identified priority areas in national development plans for which there was reasonably precise subnational aid information to expect demand and use. In turn, the focus sectors informed the selection of which government, development partner, and civil society leaders were most relevant to assessing the role of data to inform decisions on development finance projects based upon their responsibilities and expertise.

In identifying key informants for this research, we leveraged the sampling frame from AidData’s 2014 Reform Efforts Survey of 55,000 public, private, and civil society leaders in 126 low- and middle-income countries to select a subset of the most relevant decision-makers for this study.13 We then conducted desk research to further refine to identify additional institutions, positions, and contacts relevant to our research questions. Finally, we used snowball sampling to identify additional interviewees. The research team ultimately conducted in-person, semi-structured interviews with nearly 200 senior and mid-level leaders based in Dakar (Senegal), Dili (Timor-Leste), and Tegucigalpa (Honduras) in early 2016.

In the context of this study, we define development data broadly, including data on: inputs, outputs, and outcomes.14 However, we prioritized categories of data that are most relevant to decisions related to targeting, monitoring, and evaluation of development finance. We include a consideration of both open and proprietary data of various kinds (e.g., national statistics, project-level results data, budget and expenditure data). Interviewees spoke to their use of both raw, tabular data as well as other program documentation or narrative reports. Since official development assistance (ODA) is a key component of the total resources available in these countries and has been the focus of much of the AidData Center for Development Policy’s (ADCP) work, we included this as a particular emphasis of this research.

Disaggregated data, particularly geo-referenced aid and results data at a local level, is also an area of special interest in this study. In their 2014 report, A World That Counts, the Independent Expert Advisory Group on the Data Revolution for Sustainable Development placed significant emphasis on disaggregated data, saying that for data to be useful “it must be of high quality [and] at a level of disaggregation appropriate to the issue at hand” (United Nations, 2014, p.14). Similarly, USAID, in a series of Aid Transparency Country Pilots, identified the lack of data disaggregation in particular as a challenge and reported that there was a revealed demand among those they had interviewed for more subnational information (USAID, 2015).

The research team recognizes that our analysis does not include a discussion of some categories of information that are still highly valuable to local data ecosystems. One clear omission is that of citizen-generated data as a highly promising source of information to augment and validate the official record. In our recommendations we see clear synergies to explore in using citizen-generated data to strengthen domestic capacity for data production, as well as increase user confidence in the credibility of government-produced development data. Citizen-generated data to support targeting, monitoring, and evaluation of projects funded by development finance is certainly an area worthy of future study and experimentation.

The remainder of this report consists of five chapters. In Chapter 2, we summarize the main themes that tie the three country studies together. We then present each of the findings and recommendations emerging from the specific country studies, including: Honduras (Chapter 3), Timor-Leste (Chapter 4), and Senegal (Chapter 5). We then conclude in Chapter 6 with a final reflection on implications and lessons learned for the open data movement and the data revolution for the Sustainable Development Goals.

Chapter 2

Production to Use: Getting More from Development Data

Authors:

Samantha Custer and Tanya Sethi

Government, development partner, and civil society leaders make decisions every day about how to allocate, monitor and evaluate development assistance.15 Which communities should receive scarce resources? Which programs generate the highest return on investment and should be flagged for expansion? Are development projects being delivered on time and achieving their desired results?

To answer these questions, policymakers and practitioners can theoretically draw from more data sources16 on a myriad of topics and in a variety of formats than ever before (Parks et al., 2015; Custer et al., 2015). But will they choose to do so? Those who collect data and produce evidence are often far removed from those who ultimately influence and make decisions.17 Moreover, technocratic ideals of evidence-informed policymaking and data-driven decision-making are easily undercut by individual prerogatives, organizational imperatives, and ecosystem-wide blind spots (Bowen and Zwi, 2005; Pew-MacArthur, 2014).18

In this chapter, we provide an overview of the development data landscape in Honduras, Timor-Leste, and Senegal, and discuss why these trends are broadly relevant in other country contexts. Second, we present a theory of change and an assessment rubric as the framework for our analysis of the enabling environment for data use to inform development decisions. Finally, we apply this framework to summarize binding constraints that decision-makers19 from the three countries face in putting development data to use. Later chapters provide more in-depth coverage of the development data ecosystem in each of the three countries.

Section 2.1
Landscape:
Who produces data, for what purpose(s), and for whom?

Prior to assessing enablers and constraints to use, we need to understand the broader development data landscape. In this section, we provide a brief overview of who produces development data, for what purpose(s), and for whom in Honduras, Senegal, and Timor-Leste. We also draw upon additional literature to discuss how these three countries are likely illustrative of common trends in other countries.

Section 2.1.1
Governments produce a wealth of administrative data and official statistics; however, much of this information is proprietary

While there is a growing diversity of data producers, host governments generate the vast majority of information on development inputs, outputs, and outcomes in their countries. Line ministries and local governments produce a treasure trove of administrative records related to the services they provide (e.g., schools, hospitals), the people who access those public goods (e.g., students, patients), and the benefits they deliver (e.g., enrollment and immunization rates). Meanwhile, national statistical offices field censuses and household surveys to collect demographic and social statistics, including sector-specific information on agriculture, health, education, and employment indicators (UNSTATS, 2001).20

Unfortunately, in the context of Honduras, Senegal, and Timor-Leste, much of this government data is still proprietary, despite broader transparency commitments.21 In the absence of a centralized (public) system, government databases are frequently accessible only to the ministries that own them or the development partners that finance them, increasing the risk of duplication. For example, development partners often report on their activities multiple times in response to different requests from line ministries, as well as submitting information to centralized aid information management systems (AIMS) overseen by ministries of finance or planning.

Siloed data systems are a common feature of the information landscape beyond the three countries included in this study. As Custer et al. (2016a) describe in an assessment of open government initiatives in the Philippines, “information islands” persist when ministries and organizations resist integration of their data holdings in order to maintain control and protect their turf. This fragmentation can substantially increase coordination and maintenance costs of disparate data systems, while decreasing the utility of development data. In the Philippines and other countries, there is also a downstream effect of fragmented information – it shields suboptimal investments from scrutiny because the data points are too difficult to reconcile.22

Section 2.1.2
Development partners subsidize host government production of administrative data and statistics, with positive and negative externalities

In the face of scarce domestic resources, many host government databases and censuses rely on external funding from development partners such as the World Bank and the United Nations. The situation in Honduras is illustrative: the UN partly funds the Central Education Management Information System (Sistema de Administración de Centros Educativos, SACE); the German government financed a database on education infrastructure; and the Inter-American Development Bank supported a census of population and housing (IDB, 2011).

Development partner support for data and statistics in not unique to Honduras, Senegal, and Timor-Leste. According to Paris21 (2016), low- and middle-income countries received $470 million in official development assistance (ODA) for national statistics in 2014 alone. These investments – relatively small compared to other categories of ODA activity23 – are critical to bolstering domestic statistics capacity and advancing aid effectiveness principles which emphasize country ownership, a focus on results, partnerships for development, and shared responsibility (PARIS21, 2016; OECD, 2012).

Nonetheless, donor support for data and statistics may inadvertently privilege external priorities over domestic demand. As Sandefur and Glassman (2014, p.24) describe in their assessment of national statistics in sub-Saharan Africa, foreign aid can substantially influence which data is collected and the methods used to capture this information, as “national statistics offices prioritize [responding to] paying customers” rather than the needs of domestic policymakers. If development partners favor standardized international methodologies and cross-national comparisons, this may explain why many of those interviewed in this study reported their dissatisfaction with the dearth of disaggregated data available to them.

Section 2.1.3
Development partners are interested in sharing information on development cooperation to curb fragmentation and improve targeting, but they often bypass official systems

Official development assistance (ODA) is an important source of financing for development activities in Honduras, Timor-Leste, and Senegal – contributing around 15% or more of the total budget of these countries.24 While development partner interviewees did affirm the desire to have visibility on the activities of other development actors – financial amounts, sectors, project descriptions, and locations – there was low awareness of formal aid information management systems (AIMS).

Initially designed as a tool for a ministry of finance or planning to track ODA, the AIMS are also envisioned as a broader public good: a centralized, one-stop-shop for granular, project-level information on development assistance in a given country. In Timor-Leste and Honduras, the AIMS are open to the public online, while access is somewhat restricted in Senegal. Yet, across the three countries, some interviewees indicated a preference for accessing the information they required from personal contacts, donor working groups, or just-in-time data collection. These insights into how people access and share information on development cooperation illustrate a broader point: social networks remain a preferred source to access a wide range of development data in many countries. Decision-makers have long-standing practices of informal information sharing that predate the introduction of centralized databases and formal information management systems.25 The allure is strong due to a combination of convenience and lack of confidence in formal systems and databases. As a result, government officials, development partners, and CSOs turn to informal professional and personal networks to gather their intelligence.

Section 2.1.4
Civil society organizations (CSOs) are an untapped resource for highly granular project-level data,as they often fund their own data collection efforts to fill perceived gaps

As implementers of development projects, domestic and international CSOs collect project-level data as they deliver front-line services (e.g., schools and clinics), provide training and expertise (e.g., agricultural extension programs), and build capacity for people to improve their livelihoods (e.g., micro-loans). Other CSOs serve as research institutions or watchdog agencies, producing relevant data on sector trends and overall development progress, as they analyze policies, monitor indicators, and ensure accountability of public sector institutions.

Increasingly, citizens and CSOs are viewed not only as consumers, but also producers of highly granular, project-level data that could be of great use to other development policymakers and practitioners (UNDP, 2011). Wilson and Rahman (n.d, p.2), in their assessment of opportunities for government and civil society collaboration on data for sustainable development, argue that, “citizen-generated data…can complement, validate, and enhance official statistics.”26

However, in the three countries included in this study and others around the world, “unofficial” data on development resources and results, such as that produced by CSOs, is seldom integrated with official data sources produced by the government.

Section 2.2
Markets for Data: Getting to use with the four C’s framework

In this chapter, we analyze the binding constraints to data use that emerged across the three countries using a theory of change and assessment rubric adapted from a Custer et al. (2016a) study, From Pork to Performance: Open Government and Program Performance Tracking in the Philippines.

The original study examines the interaction of technology, transparency, and politics in the context of improving service delivery in five sectors. The authors seek to explain how transparent data on public resources and results could theoretically “catalyze a chain reaction that mobilizes citizens, politicians, and frontline providers to connect the dots between government spending and the tangible services they experience in their daily lives” (p.xxii).

The Custer et al. (2016a) framework presents the causal logic of getting from data to impact (e.g., improved development outcomes) as the interaction of four C’s: content, channel, choice, and consequence.27 We slightly modified this framework as a theory of change for development data use in our three countries. Specifically, governments and organizations disclose data on development resources and results (content), disseminate this information to users online or offline (channel), whereby citizens and officials take action individually or collectively (choice), to improve the country’s performance on achieving sustainable development for all (consequences).

On the surface, this theory of change appears to be a straightforward story of supply (governments and organizations produce and disseminate data) and demand (policymakers and practitioners want this information to use it to improve how they allocate resources and monitor progress). In reality, the narrative is much more complex. Each step in our theory of change involves choices underpinned by tenuous assumptions. The wider the gap between those producing and using development data, the greater the risk of a mismatch between supply and demand (Custer et al., 2016a and 2016b; Development Gateway, 2016).

For example, if development data is to be put to use, the information must be timely and salient to end users (content), easy to access and use (channel), accompanied by credible outlets for people to act upon it (choice), and their action(s) must be sufficient to change how policies are designed or programs delivered (consequences). Unfortunately, for producers and users, aligning supply with demand for development data more often than not requires substantial trial and error.

Data Use: What are the binding constraints?

In this section, we summarize the common constraints to data use in Honduras, Senegal, and Timor-Leste using the framework adapted from Custer et al. (2016a). In the context of this study, we exclusively look at the first three C’s as building blocks for data use. We also discuss how these data use constraints are likely to manifest in other countries. For additional information, Chapters 3, 4, and 5 provide a deeper exploration of the enablers and constraints to data use in each country independently.

Section 2.2.1
Content

The first building block in getting to data use is related to whether development data itself is “fit-for-purpose.” In other words, prospective users of this information must deem it to be appropriate, or well suited, to answering their questions.

As we discuss below, these supply-side failures are a major deterrent to use of development data across stakeholder groups.

Our interviews with government, development partner, and CSO participants revealed several content constraints, namely, that development data is often:

  1. Insufficiently Granular
  2. Of questionable accuracy; and
  3. Fragmented across numerous organizational and agency silos
Section 2.2.1.1
Granularity: Government, civil society, and development partner leaders want more data that is disaggregated by sector, geography, and demography

There was a groundswell of interest across government, development partner, and CSO interviewees in having more data on development infrastructure (e.g., clinics, schools) and results (e.g., student performance, unemployment figures) that is disaggregated by district (or relevant subnational unit). Yet, demand for this information outstrips the capacity of the three countries to produce it consistently. When asked about high-value data sources, interviewees most frequently mentioned geo-referenced and sector-specific administrative data produced by line ministries, and surveys and censuses from national statistical offices.28

In Timor-Leste, for example, respondents expressed interest in greater disaggregation of survey and census data by district, subdistrict, and village (three administrative levels).29 In Senegal, development partners specifically requested disaggregated information on crosscutting issues such as gender parity, climate change, and governance, as well as other sector-specific indicators. In Honduras, interviewees across stakeholder groups reported interest in municipal-level socio-economic data and sector-specific indicators, such as: out-of-school children, topographic conditions for crops, and school-based violence.

This interest in disaggregated information is consistent with reports from other countries and international experts. Analyzing preliminary usage patterns from Kenya’s Open Data portal, Rahemtulla et al. (2011) identify a high degree of citizen interest in more granular or localized information. USAID (2015), in a series of Aid Transparency Country Pilots, highlights the lack of data disaggregation as a specific challenge and reports demand among those interviewed for more subnational information. The UN (2015, p.11) notes in its reflection on the Millennium Development Goals (MDGs) that having more, high-quality data disaggregated across various dimensions (e.g., age, sex, ethnicity, disability, geography) will be “key to making decisions and monitoring progress in achieving sustainable development for all.”

If disaggregated data is in short supply relative to demand, this is a missed opportunity to encourage broader use of government data. This supply-side failure is a likely by-product of the fact that the production of highly granular sector-specific data relies on providers at the point of service delivery (e.g., school administrators, medical professionals, agricultural extension workers). Front-line workers often have constrained data skills to report information accurately, and limited visibility on its value if they do. This dynamic creates a stunted feedback loop with weak incentives for front-line providers to change the status quo (Custer et al., 2016a).30

Section 2.2.1.2
Accuracy: Development data sources are frequently viewed as incomplete or out of date due to technical capacity constraints and perverse political incentives

Governments, particularly national statistics offices, are among the most prolific data producers in low- and middle-income countries. Unfortunately, concerns regarding the timeliness, accuracy, and validity of government-produced data substantially dampened broader use among those we interviewed. While most often associated with government data, these challenges extend beyond the public sector. For example, development partners often do not allocate sufficient attention to reporting on their activities in a timely manner, hence concerns raised in all three countries regarding the timeliness of data published via aid information management systems.

Interviewees cited several technical constraints that compound the risk of inaccurate data, including: publication delays, episodic data collection, limited data management capacity at the subnational level, and gaps in coverage. In Timor-Leste, front-line service providers are under-resourced, under-trained, and under-motivated to submit complete and accurate reports. In Honduras, financial and technical constraints were cited as the primary reasons why data reported by local staff is inaccurate. In Senegal, school and clinic-level data is still collected via pen and paper before digitization at the district level.

While data producers grapple with substantial technical capacity gaps, interviewees additionally emphasized that there are also political reasons driving inaccuracies in development data. Interviewees noted that powerful incentives exist for public servants to make the official numbers look good.31

Inaccuracies in official data sources, whether intentional or unintentional, are not unique to Honduras, Senegal, and Timor-Leste, however. Devarajan (2011, para 8) acknowledges that “weak capacity in countries to collect, manage, and disseminate data” is a contributing factor, but points to politics as the underlying driver to explain the poor state of statistics in Africa. Election cycles create domestic pressures for politicians to falsify statistics to garner the support of constituents (Devarajan, 2011). Development partners can also inadvertently create perverse incentives for officials or front-line providers to misreport on their country’s progress through tying performance incentives to meeting certain targets as reported via administrative data the government collects (Sandefur and Glassman, 2014).32

Unfortunately, as most government ministries have low data management capacity (particularly when it comes to procedures for data verification and auditing), there are limited internal safeguards to mitigate this risk of data manipulation, which further lowers user confidence. In response, development partner and CSO respondents resort to fielding independent data collection efforts to supplement or circumvent the use of official data. These unofficial data collection efforts are a missed opportunity to validate and augment official data sources. As Sandefur and Glassman (2014, p.24) argue, “getting administrative data and [independent] survey data to speak to each other” can create powerful checks and balances to improve accuracy.

Section 2.2.1.3
Silos: Development decision-makers need to triangulate diverse data points, but disconnected data initiatives make integration and interoperability difficult

Whether they are making new investments, monitoring current progress or evaluating past performance, decision-makers need to see the full service delivery chain from upstream budgets and intermediate outputs to downstream outcomes. Unfortunately, fragmentation is often the norm in Honduras, Senegal, and Timor-Leste, as it is in many other parts of the world: disconnected data points are siloed between disparate information management systems maintained by different agencies and collected using non-standardized methods (Custer et al., 2016a). As a result, decision-makers and their support staff must expend significant time, money, and effort to put this information to practical use.

Having launched an ambitious national development strategy in 2015, the administration of Senegal’s President Macky Sall has since emphasized the role of data to guide sectoral priorities and budget allocations. The Honduran government, meanwhile, is developing a centralized results-based management system to integrate data across ministries on inputs, outputs, and outcomes. To succeed in these efforts, public sector leaders will need to address head-on the intractable issues of integration (data sharing) and interoperability (data harmonization).

These challenges extend beyond the government. Development partners and CSOs expend substantial resources to produce development data that meets their specific needs. These independent data collection exercises often use different methodologies and generate datasets that may not be publicly available, making comparisons difficult. In Timor-Leste, several interviewees noted the substantial risk of reinventing the wheel with ad hoc data collection efforts that are seldom coordinated with each other, even within departments of the same organization. One development partner reported that they were conducting 3-4 different household surveys to collect slightly different data from the same population.

Data silos not only are costly to maintain and frustrating to navigate, but are also viewed by several interviewees as likely to increase the risk of data gaps and errors.33 The United Nations (2015) shares this perspective, arguing that integration of data sources and adoption of standardized methods and formats benefits data producers and consumers alike by reducing the likelihood of errors, long-term costs, and respondent burden (e.g., participant fatigue or partial compliance in response to multiple data requests).

In the face of consequential capacity gaps, governments, development partners, and CSOs could enjoy gains from a more concerted effort to make their data sources comparable across sectors, geographies, and time periods, as well as through shared training and technical staff. However, it is clear that connecting information islands will require political leadership to convince competing agencies and independent organizations that the tangible benefits of adopting common data formats, re-engineering data collection processes, and changing institutional arrangements outweigh the perceived pain points (United Nations, 2015).

Section 2.2.2
Channel

The second building block in getting to data use relates to the channels that producers of development data employ to disseminate it to prospective consumers. To what extent is information publicly available or privately held behind firewalls or paywalls? Is development data available only to the digitally savvy, or is it also accessible offline? Are target users aware that the development data exists and are they able to easily find it?

Interviewees from all three stakeholder groups (governments, development partners, and civil society organizations) reported three ways in which these dissemination channels miss the mark:

  1. There is uneven access to information;
  2. Different levels of connectivity exacerbate a digital divide; and
  3. If data is invisible, people will not find and use it.
Section 2.2.2.1
Access: Absent Freedom of Information legislation, development data access depends upon commitments to voluntary disclosure

The rise of information communication technologies is decreasing the cost, time, and effort to produce a wealth of development data; however, greater availability of information does not necessarily translate into equitable access. In Timor-Leste and Senegal, both countries without Freedom of Information (FOI) laws, visibility on public sector activities depends on the government’s willingness to open up this data, or on the strength of one’s network to secure it via back channels. This reliance on voluntary disclosure creates uneven access that disproportionately affects less well-connected groups and individuals. In Senegal, civil society groups cannot readily access the national statistics agency’s data and civil servants struggle to obtain hyper-local, disaggregated data managed by other ministries.

Even in countries that have adopted FOI legislation, access to development data is only as strong as the government’s commitment to uniformly enforce compliance. In Honduras, much of the sector-specific indicator data in demand is either not available or not open, despite high-profile transparency commitments.34 In the agriculture sector, the government recently launched an internal government platform for monitoring agricultural outcomes, which is not public. The national statistics agency charges a fee for some of its household survey data and the Ministry of Education’s database of education indicators35 is not yet entirely publicly available.

Development partners and CSOs are also often reluctant to share information and they typically do not fall under the jurisdiction of FOI law. For example, interviewees report that development partners and CSOs conduct independent data collection exercises and maintain proprietary databases that have valuable information on development projects and indicators, but are not in the public domain. Some central government officials, that view development partners and CSOs as bypassing them to collect information directly from local government officials without sharing their results, also hold this perception.

Unfortunately, this mismatch between supply and demand is not unique to the three countries in this study. When governments exempt certain categories of information or types of agencies, they dilute the power of FOI laws. Meanwhile, voluntary disclosure practices, such as open data initiatives, are often the product of compromise (Gigler et al., 2011; Custer et al., 2016a). The datasets that governments and organizations make available are the easiest to publish, not the ones most in demand (Davies, 2014). A government or organization often chooses which data to make available in isolation, rather than through a process of gauging public interest in various data (Canares and Shekhar, 2015).

In fairness, government agencies, development partners, and CSOs often face dueling imperatives: paying for large-scale data collection and management infrastructure using their own resources, while still answering the call to make data freely accessible and open to all. Advocates for transparency and open data have arguably been more successful in generating political pressure for openness than in helping governments and organizations grapple with the sticky issues of business models to cover the costs of data collection and management sustainably. In addition, as Custer et al. (2016a) describe in an assessment of the Philippines’ open government initiatives, data producers are also concerned about non-monetary costs, such as citizens misinterpreting information or accusing them of misleading the public if data is inaccurate.

Section 2.2.2.2
Connectivity: Delivering development data exclusively via online channels makes access difficult due to connectivity and human capacity constraints

The proliferation of smart phones and increasing Internet penetration in the past decade has done much to democratize data and reduce information asymmetries. However, there remains a profound divide between those that are part of this digital revolution and those left behind (World Bank, 2016; Gigler et al., 2014). Limited Internet connectivity, language barriers, and low levels of data literacy and numeracy inhibit practical access to, and use of, development data in many countries and stakeholder groups.

The case of post-conflict Timor-Leste is particularly poignant. In multilingual Timor-Leste, development data is often available online in the official languages of Portuguese and/or English, but not in Tetum, the lingua franca of most citizens.36 In addition, a weak educational system has led to a dearth of professionals trained in literacy, numeracy, and computer skills. As a result, domestic policymakers and practitioners are predisposed to favor an oral culture of information sharing and are far less comfortable using complex online databases.

Even among the 1.1% of Timor-Leste’s citizens that use the Internet (World Bank, 2012), interviewees report that slow and intermittent connectivity makes it difficult to share information via online channels. For example, one line ministry reportedly developed a parallel system to collect data on development cooperation activities via Excel spreadsheets emailed to users that lack a stable Internet connection. Other respondents resort to calling contacts to access online systems on their behalf, or using the in-person development partner working group meetings as a workaround to get the information they need.

Connectivity and capacity constraints in utilizing online sources of development data are not unique to Timor-Leste. If government, development partner, and CSO producers of development data rely exclusively on online dissemination strategies, this can inadvertently privilege access to information among the urban, educated, and well-to-do (who tend to be more digitally savvy) at the expense of other groups (Read and Atinc, 2017; Custer et al., 2016a).

Section 2.2.2.3
Invisibility: Prospective data users have low familiarity with many publicly available information sources

Even when data is available and accessible, it may not be visible enough to prospective users for them to take advantage of this information in their work. Nonetheless, government, development partner, and CSO producers seldom have an intentional strategy to proactively communicate and promote the development data they publish via datasets, databases, and data portals. For example, development partner representatives interviewed in Honduras, and even fewer CSOs, were aware of publically available sources of development cooperation data, such as the country’s open aid information management system. People will not use development data that they do not know exists, increasing the risk of data graveyards.

A similar narrative emerges from other data initiatives beyond this study. In an assessment of governance data use in 126 low- and middle-income countries, Masaki et al. (2016, p.3) note that producers rarely have a “proactive strategy to conduct outreach with domestic media outlets” even though media is “one of the top ways users say they learn about new governance data sources.” Similarly, Custer et al. (2016a) find that in opening up information on public service delivery, the Government of the Philippines and the World Bank put substantial time and effort into connecting disparate data islands, but paid comparatively little attention to promoting the existence of the public data portals. Advertising was often limited to a one-off launch event and/or ad hoc training events with local journalists.

Section 2.2.3
Choice

The third building block in getting to use is one which producers of development data have comparatively less control over than the first two C’s: will people choose to act upon this information, either individually or collectively? In their internal calculus, potential users weigh the prospective benefits to themselves and their institutions versus the costs of putting development data to work in their decision-making. There is an extensive literature on the myriad factors that decision-makers may consider when deciding whether to use a particular information source.

Our interviews with government, development partner, and CSO participants highlighted three constraints that rose above the rest:

  1. The development data inquestion lacks credibility
  2. Prevailing norms of information sharing via informal networks rather than official sources aredifficult to overcome; and
  3. There are perverse incentives to crowd out data and letother factors drive decisions.
Section 2.2.3.1
Credibility:
Government-produced data suffers from a trust deficit, with many respondents viewing these data sources as out-of-date and unreliable

As is often the case in low- and middle-income countries, government-produced data in Honduras appears to suffer from a trust deficit, which undercuts use. CSO representatives in particular raised concerns that mayors and other officials sometimes skew data to receive more funding (e.g., adjusting population figures) or under-report bad news to paint a rosier picture than reality. Interviewees raised examples such as: teachers abetting students to cheat to boost test scores, nurses not registering people with Chikungunya (a mosquito-borne illness) to lower the disease incidence rates, and the lack of sound population data making it difficult to calculate the percentage of children in school.

Section 2.2.3.2
Incentives: Data is only one part of the calculus, and leaders may still ignore evidence in favor of convenience or organizational imperatives

As development policymakers and practitioners make decisions about which programs and projects to fund, they inevitably create both winners and losers. In this respect, resource allocation decisions are likely fraught with more political jockeying than the design or evaluation of the resulting projects. This is consistent with the experience of interviewees across the three countries who report that upstream budgeting and planning processes are substantially shaped by existing strategies, organizational imperatives (e.g., headquarters’ directives, tied funding), or political exigencies (e.g., protecting interests of allies, alleviating concerns of vocal constituencies).

Some government, development partner, and CSO representatives across the countries reported using data in the context of siting new projects (e.g., feasibility studies) or assessing project results at closure. However, even at these parts of the project cycle, development partners and CSOs make decisions from more than the evidence alone, as they may select project locations based on ease of access, available infrastructure, and existing networks. Political pressures or the desire to maintain good working relationships can also undercut willingness to frankly discuss (or publicize) lackluster results in implementation, thereby impeding organizational learning and corrective action.

Beyond organizational factors, societal norms and individual biases can also decrease the perceived value of drawing upon data and evidence throughout policymaking processes or project cycles. In Timor-Leste, for example, interviewees emphasized the country’s strong oral culture and indicated that government officials often have an unfavorable view towards using data and evidence because they associate this information with international experts.

When government, development partner, and CSO representatives choose to use data and evidence in their decision-making, they pay the costs upfront in time, reputational risk, or lost political capital for uncertain future rewards. A Development Gateway (2016, p.3) assessment of results data use emphasizes that there is seldom an explicit requirement that development decision-makers in low- and middle-income countries base their decisions upon evidence, and few rewards if they do so. In fact, data analysis and evidence-informed policymaking is viewed as “an overtime activity…[only] for the intrinsically motivated.”

The absence of clear rewards for making decisions based upon evidence, and of penalties for not doing so, gives tacit permission for policymakers and practitioners in all three countries to pay lip service to the importance of development data, while allowing other considerations to crowd out using this information to change the ways decisions are made.41 It is telling that the most frequent use case mentioned by interviewees was data for reporting purposes, namely as inputs to report on progress to development partners or the Chief Executive’s office. It remains to be seen whether results-based management and evidence-informed policymaking initiatives, such as Senegal’s effort to emphasize a greater role for data to guide sector priorities and budget allocations in the education and health sectors, will fundamentally alter the status quo.

Section 2.2.4
Learning from Development Leaders in Three Countries

The crosscutting themes in this chapter and the country-specific findings presented in the next three chapters have far-reaching implications for the ongoing sustainable development data revolution discussion and debate (United Nations, 2014). In Chapter 6 we will conclude with recommendations for those who wish to design data solutions that are responsive to revealed demand and remedy the binding constraints that undercut data use.

Producers, users, and funders of the next generation of development data will need space to experiment with new approaches, test assumptions, and document insights. Nevertheless, the revolution can get a helpful head start by learning from the first-hand experiences of decision-makers in Honduras, Timor-Leste, and Senegal regarding whether and how they use the development data presently available to them – and why.

Chapter 3

Translating Transparency into Action for Sustainable Development in Honduras

Authors

Tanya Sethi, Carmen Cañas, Jose Sierra David Castillo, Caleb Rudow, Deirdre Appel

To become more transparent and effective in delivering value for its citizens,42 the Government of Honduras (GoH) has opened up its financial management to public scrutiny and created a legal mandate for access to information. 43 Yet it remains to be seen how publicly accessible data on government investments and results will translate into improved resource allocation and service delivery.

This study examines the prospects and limits for development data to help public, private, and civil society actors in Honduras monitor and measure progress towards the country’s goals. What data is available on development resources and results? How are decision-makers in Honduras using this information and for what purposes? Our research team put these questions to the test through interviews with 68 senior and mid-level decision-makers from government agencies, development partners, and civil society organizations (CSOs).44

With a focus on the education and agriculture sectors, we asked these prospective data users about the types of information that are most useful in their efforts to allocate resources, monitor results, and evaluate progress against development goals. These users also helped identify critical barriers to data use and opportunities to close the gap between demand and supply.

Section 3.1
Landscape: Who produces data, who uses it, and why?

What role does data play in the multitude of decisions that the Government of Honduras, development partners, and CSOs must make every day? In this section, we discuss the data produced by each of these stakeholder groups, examine different use cases, and identify the types of data in greatest demand. We pay particular attention to information on official development assistance (ODA) – the distribution of dollars and project results by sector and location – as a crucial part of the data ecosystem, given the importance of external grants and loans to Honduras’ financing for development.

Section 3.1.1
Government of Honduras

Several government agencies produce development data; however, not all of this supply is financed domestically, as is the case in many developing countries. In 2009, for example, the government cut the budget of the National Statistics Institute (Instituto Nacional de Estadística, INE) to curb the country’s deficit. This, in turn, increased the INE’s dependence on external financing to field surveys, create official statistics, and maintain databases. Production of development data in Honduras remains anemic, for decades limited to collecting basic economic data such as the country’s gross domestic product, balance of trade, and tax revenues.

Section 3.1.2
Development partners subsidize the government’s official data production

Development partners recognize that strengthening the government’s national statistical capacity – the ability to produce reliable and timely information on development inputs, outputs, and outcomes – is critical. A greater investment in domestic capacity for data production is necessary to realizing the principles of country ownership, a focus on results, partnerships for development, and shared responsibility that are codified in the 2011 High-Level Forum on Aid Effectiveness.

As a result, many government databases and the national census of population and housing in Honduras receive external funding for their implementation and maintenance from development partners, such as the World Bank, the German government and the United Nations.45 However, dependence on foreign aid may affect the country’s ability to build sustainable institutional capacity for statistics in the long run. Moreover, this status quo may inadvertently privilege external priorities over domestic demand when it comes to identifying what types of data collection or capacity to finance.46

Section 3.1.3
Most government data is proprietary, despite broader transparency commitments

The vast majority of government-produced development data is proprietary, maintained by line ministries or the President’s Office (see Table 1). Databases are often only accessible to the government agencies that own them or the development partners that finance them. Even within government, there is no centralized system to access different public sector data sources.

If there is a huge investment in collecting data and this is not at the disposal of the public, what is the point?
Government Respondent

The National Statistics Institute (INE) is an important source of publicly available information (some of which is open). The INE publishes much of the underlying data and analytical reports from its household surveys, the national census, and the agricultural census – important sources of publicly available information.47 Honduras’ aid information management system (Plataforma de Gestión de la Cooperación, PGC) is another valuable source of open data. Hosted by the secretariat of external relations (Secretaría de Relaciones Exteriores, SRECI), the PGC publishes data on the distribution of development assistance projects by location, sector, and programmatic area.48

Section 3.1.4
The Government of Honduras uses development data more for reporting, less for decision-making

The Organization for Economic Cooperation and Development (OECD, 2009) notes that for many years policymakers in Honduras have rarely used available statistical information to make decisions. Decision-making regarding national development priorities, programs, and resources follows a largely top-down process that is led by the central government and features only limited interaction with subnational governments.49 CSO respondents reported that the municipal leaders lack autonomy to make their own decisions, which limits their role to implementation and reporting back to the capital.

Government respondents in this study most commonly cited reporting on performance to the Presidential Office or development partners as an important use case for development data. For example, in 2014, President Hernández instituted a centralized results-based management system (Sistema de Gestión por Resultados, SGPR) to integrate data on inputs, outputs, and indicators from various government ministries in order to streamline monitoring of service delivery.50 Yet, government respondents stated that they produce this data primarily to report on indicators requested by the President’s Office and the secretariat of general coordination (Secretaría de Coordinacíon General, SCGG) to evaluate progress against the National Plan.51

Government users of Honduras’ aid information management system, the PGC, tell a similar tale. Originally envisioned by the government as a planning and coordination tool, current use of the PGC is largely confined to bilateral meetings between the secretariat of external relations (SRECI) and development partners to review projects and verify information in the platform for reporting purposes. Previously housed under the secretariat of planning and external cooperation (Secretaría de Planificación y Cooperación External, SEPLAN), the PGC was once more closely integrated with the national planning system. According to some respondents, the political influence of the PGC was diminished when a new administration transferred responsibility for the system from SEPLAN to the SRECI in 2014 as part of a broader government restructure.

While reporting is the most common use case, government respondents shared anecdotes of instances when they have used data to make course corrections and improve targeting. For example, government representatives reported using a database produced by the education secretariat52 and demographic data from the INE to identify a gap between the number of five-year-olds living in Honduras and those enrolled in primary school. The Government of Honduras subsequently launched a successful national program to bolster five-year-old enrollments. Others also cited these data sources as being useful in their work to identify under-performing schools and compute other indicators.53

Table 1: Government Data Producers and Development Information Sources in Honduras

Data Producers

Information Sources

Open Data?

Secretariat of General Coordination

Secretaría de Coordinación General (SCGG):

Responsible for strategic planning, including: allocation of resources and identification of sector-specific development targets

Results-Based Management System

Sistema de Gestión por Resultados (SGPR):

A personalized database of socio-economic data

National Social Sector Information Center Database

(Centro Nacional de Información del Sector Social, CENISS):

Four databases with data on return migrants, persons eligible for social programs, children aged 0-6 who receive social benefits, and characteristics of social projects

No

Secretariat of Finance

Secretaría de Finanzas (SEFIN):

Prepares the annual budget and determines budget ceilings for external loans and expenditures. Provides technical assistance and monitoring for ministries in the budget preparation process

Integrated Financial Management System

Sistema de Administración Financiera Integrada (SIAFI):

A centralized database of all financial government data

Integrated Municipal Administration System

Sistema de Administración Municipal Integrado (SAMI):

A financial database at the municipal level; an initiative of the national government, it has been rolled out to 40 out of 298 municipalities as of May 2016

No

Secretariat of External Relations

Secretaría de Relaciones Exteriores (SRECI):

Coordinates, implements, and monitors the GoH’s foreign policy, including relationships with development cooperation partners in Honduras

Aid Information Management System

Plataforma de Gestión de la Cooperación (PGC):

Database that tracks the distribution of development cooperation projects in Honduras, including: financial, programmatic, sectoral, and location data

Yes

Line Ministries / Sector Secretariats:

Executes the GoH’s Strategic Plan, supported by secretariats and institutions with their own Planning and Management Evaluation Units (UPEGs) and directorates

Central Education Administrative System

Sistema de Administración de Centros Educativos (SACE):

A school-level database maintained by the Ministry of Education with information on students and teachers in the public education sector

Partial

National Institute of Statistics

Instituto Nacional de Estadística (INE):

Produces official statistics to support the GoH and others in planning, budgeting, and evaluation of development activities

INEbase:

National socio-economic statistics produced by the INE, including: household surveys, national census, agricultural survey

Partial

Section 3.1.5
Development Partners

The devastation wrought by Hurricane Mitch in 1998 created a spike of external assistance flows into the country.54 Almost two decades later, Honduras continues to rely heavily on this external support. In 2015, official development assistance (ODA) made up 18% of the government budget.55 As a result, a large community of multilateral and bilateral development partners advises the government on policy and proposes projects.56 Development partners also finance data collection activities, either for their own use or to complement the government’s efforts.

Section 3.1.5.1
Development partners produce data and evidence that is broadly perceived to be reliable and useful

Development partners create databases and specialized reports, fund data collection by the government, build capacity through initiatives to strengthen data systems, and publish information about their interventions in evaluation reports. Most do not collect detailed project-level data on monitoring themselves, working instead through their implementing partners (e.g., CSOs or contractors). However, development partners often hire foreign consultants to undertake these activities rather than working through the government or local CSOs, which effectively deters building a strong cadre of domestic data scientists and statistics professionals.57

Nonetheless, development partners produce indicator data that all stakeholder groups find to be reliable and useful. One frequently cited example was the United Nations Development Programme’s Human Development Index (HDI). The secretariat of external relations uses the HDI in its aid information management system (PGC), and development partners leverage the data source to perform country assessments. Meanwhile, CSOs find the HDI helpful to justify interventions in the least developed municipalities of the country.58 Interviewees also mentioned macro-statistics from the International Monetary Fund and poverty maps produced by the World Bank and the Inter-American Development Bank as data sources that were useful in their work.

Development partners publish sector-specific analytical reports to inform strategic planning efforts and evaluate in-country work.59 CSOs also identified development partner-produced analytical reports as particularly useful to help them advocate with the government for policy changes or develop stronger systems. For example, one CSO cited a World Bank report on money lost to absentee teachers as their inspiration to create a system to verify teacher attendance and advocate with the government to be more conscientious in tracking this information.60

Section 3.1.5.2
Information sharing is key to coordination, but happens behind closed doors

The government has an important role to play in curbing fragmentation and ensuring alignment of development partners with national development priorities. In 2014, more than 30 development partners were operating in Honduras, most contributing a small amount of funds to the country (OECD, 2016). Larger agencies do attempt to coordinate with each other, but there are perverse incentives among development partners and NGOs to advance their own agendas at the expense of coordination. Development partners do report using the G-16 group of the 16 biggest donors in Honduras as a forum to share information with each other.

Typically, development partners create broad multi-year and country-specific strategic plans that outline their geographic and sectoral focus areas. Often, development partners time these country plans to coincide with a presidential election, and ensure alignment with the incoming administration’s priorities and new Strategic Plan. Government and development partner collaboration varies by sector, with interviewees reporting that they work together closely in the education sector.

For example, the International Cooperating Partners’ Education Round Table (Mesa Redonda de Cooperantes Externos en Educación, MERECE) serves as a forum for development partners to share information on their projects, and for the government to present its priorities and financing needs.61 The sector roundtables – mesas sectoriales, tripartite bodies that are part of the G-16 and comprise government, CSO, and development partner representatives – are theoretically another venue for dialogue. However, their success depends upon the organizing development partner’s ability to ensure participation and continuity between administrations.

While development partners produce formal sources of data and evidence, they also rely heavily on informal back channels to get information and insights from each other on development cooperation activities. The sector roundtables allow development partners to exchange information on sector-specific projects and locations with their counterparts. Nonetheless, these venues have varying levels of participation and success, incentivizing development partners to turn to their personal connections and networks to access information on a just-in-time basis. One possible interpretation of this: development partners may view their personal connections or professional networks as complementing publicly available data sources, providing additional context in the form of lessons learned and experiences working with certain partners or communities.

Standardization is also a problem of disrupting the status quo and the country politics. People use the data to their advantage and those who control it like the power that it gives them. All these things considered, the government wants to improve on this [but] it is difficult and costly to do so.
Development Partner Respondent

Section 3.1.6
Civil Society

Civil society organizations ( CSOs) in Honduras are most prominent in the later stages of the policy process: implementing projects at the local level, monitoring public management of resources, and advocating for course corrections. Whether for the purpose of aligning with national development priorities or accessing funding, CSOs often seek to align their activities with government plans and development partner country strategies. Some CSO respondents admitted that this is a necessity, as many are heavily dependent on development partner funding, either directly or indirectly through projects channeled through the government.62

Section 3.1.6.1
Civil society organizations are an untapped resource for highly granular project-level data

CSO-produced development data could complement official data, especially at the local level. A critical part of the local data ecosystem, CSOs generally produce three types of information: baseline social data from the communities where they work, monitoring and evaluation data from the projects they implement, and reports by consultants on specific subjects. CSOs collect hyper-local data that could be a highly valuable source of information to help fill persistent gaps in official data collected at the subnational level. Yet, broadening the availability and use of CSO-produced data may be impeded by the fact that these data collection exercises often begin as a workaround. In the absence of reliable, comprehensive, and granular data from government channels, CSOs seek out alternatives to capture their own disaggregated data on development resources, activities, and outcomes. This project-level data production can be costly and time intensive, particularly for smaller organizations.

Section 3.1.6.2
Civil society organizations come in different shapes and sizes, but offer similar examples of data use

CSOs use development data in a variety of ways to inform their strategies, plans, and activities. For example, CSOs often use baseline socio-economic indicator data produced by development partners to select project locations or monitor changes over time and report on results. Large CSOs have more established ways to coordinate and share information when targeting communities to ensure no duplication of efforts.63 Smaller organizations, in comparison, typically rely on phone calls and on-site visits. CSOs also make use of thematic meetings and conferences organized by development partners to interact and exchange information with others in their sector.

[Our organization] aligns with the national plan of Honduras, because donors align with the national plan. Getting money from donors is easier if you have similar goals to the national plan.
CSO Respondent

Section 3.2
Data Use: What are the binding constraints?

Monitoring progress on national priorities and global goals requires strong statistical capacity and political commitment to produce, disseminate, and use data. However, there was a common perception among those interviewed that several barriers exist that mitigate against greater use of existing data for coordination, decision-making, and policy change.

In this section, we discuss the binding constraints to data use in Honduras by adapting a theory of change previously developed by Custer et al. (2016) and presented in Chapter 2 as a common lens to view trends across the three country studies. This approach presents the causal logic of getting from data to impact (e.g., improved development outcomes) as the interaction of four C’s: content, channel, choice, and consequence.64 Given the scope of this study, we focus on the first three C’s as foundational building blocks.

Most data and studies that are done at the local level are just thrown away once the mayor or the government changes.
Government Respondent

Section 3.2.1
Content

Government agencies, development partners, and civil society organizations (CSOs) produce and disclose a wide variety of data on development resources and results in Honduras. However, the extent to which this data is fit for its intended purpose depends upon key assumptions regarding the accuracy and consistency of the data being produced, as well as its perceived salience and interoperability.

Section 3.2.1.1
Granularity: Disaggregated and geo-referenced data is in high demand, but low supply

There is strong demand across stakeholder groups for socio-economic data and sector-specific indicators at the municipal-level, such as: out-of-school children, topographic conditions for crops, and school-based violence. The National Statistics Institute (INE) was frequently mentioned as an important source of this disaggregated data, which is often used as a base to construct other indicators. However, there was an expressed desire on the part of government, CSO, and development partner representatives alike for the INE to produce more up-to-date, accurate, and publicly accessible data.

For example, census and household survey data collected by the INE is obsolete – the last agricultural census was in the 1990s – and lacking in coverage for certain groups (e.g., indigenous populations). In addition to financing constraints, gang violence and political opposition makes the work of enumerators challenging in some municipalities.65 Government ministries and development partners sometimes undertake supplemental data collection efforts on an ad hoc basis for specific needs; however, this is insufficient to provide a complete, comparable subnational dataset.66

Information costs a lot to generate, and the ones that generate it don’t want to share.
CSO Respondent

A federal initiative, the integrated system of municipal administration (Sistema de Administración Municipal Integrado, SAMI), endeavors to capture municipal-level budget, tax, and expenditure data but includes information for only 40 out of 298 municipalities, as of May 2016.67 Municipalities also report on development projects and outcomes in their jurisdiction; however, national-level officials sometimes perceive this locally reported data to be inadequate or inaccurate. This disconnect may partly be attributed to the fact that the local government officials collecting downstream results data are less directly involved in upstream allocation and planning decisions and have little insight into how this information is used (Development Gateway, 2016).

Section 3.2.1.2
Silos: Disconnected data initiatives undercut the accuracy and utility of development data, and increase the costs of reporting and maintenance

Stakeholders frequently identified difficulties in dealing with fragmentation: disconnected data points, silos between disparate information management systems, and maintenance by different agencies. This status quo hampers the ability of even government officials to share information between ministries. Several databases are only accessible to the ministry owner, and there is no centralized system to access different government data sources.

Yet, integration and standardization are perceived to be prohibitively difficult and expensive. This creates perverse incentives for ministries to duplicate efforts and capture similar information. For example, the Government of Honduras maintains two systems that contain information on foreign aid — a financial management system used for primarily on-budget projects (SIAFI) and an aid information management system for planning and coordination (PGC). These systems may serve different functions, but sharing data could create effiency gains.68

Disconnected systems also impede the ability to leverage complementary data from multiple agencies. A new results-based management system (SGPR) being piloted by the government is an improvement over the status quo, as it will pull together data from all existing government databases. However, this is currently envisioned to be accessible only to the Office of the President with the focus on monitoring indicators in the national plan. However, the SGPR could be a unique opportunity for the government to share this information and connect the dots along the entire service delivery chain, from data on resources (e.g., domestic budgets, grants, external loans) to outcomes (e.g., socio-economic outcomes, project results).

These challenges also extend beyond the government. Poor coordination between government agencies, development partners, and CSOs leads to inefficiencies in data collection. Indicators are measured using different methodologies, a particular challenge in using baseline data.69 Datasets are not easily comparable over time, as development partners often bring in different consultants to work on the same subject over the years, each with their own ideas about how to measure development indicators. This problem is accentuated by staff turnover within development partner offices, which often means that there is little institutional memory.

Section 3.2.1.3
Capacity: Human, technical, and financial constraints perpetuate inefficient data supply that is out of step with user demands

The electoral cycle has an unintended consequence on the government’s capacity to produce development data: a loss of trained personnel and the resources spent on training them to collect, curate, and maintain data. As new administrations take office every four years, each comes with its own priorities. The resulting reorganizations can displace experienced staff to other government offices or out of government entirely.70 These capacity gaps are interlinked with financial and technological constraints.

External resources still largely fund national data capacity, creating islands of excellence in line with development partner priorities. As a result, some government institutions have better data systems than others, and the consistency of data collection varies. For example, the National Statistics Institute (INE) did not conduct a national census after 2001 or an agricultural census since 1993, partly due to lack of funds. The 2012 census was financed by the Inter-American Development Bank (IDB). Almost all respondents in the agricultural sector mentioned the lack of a recent census as a problem. Lacking another source of information in the agriculture sector, a CSO participant mentioned using the 1993 agricultural census for their current analysis.

Meanwhile, in the education sector, teachers in rural areas have limited Internet access, for example, and need to travel to towns to be able to enter the data into centralized systems. Government respondents in the agriculture sector highlighted a lack of funding, technical personnel, and technology and new techniques for recording more precise data.71

Section 3.2.2
Channel

Beyond content, creating an enabling environment for data use also requires thoughtful attention to the channel(s) used to disseminate this information and engage with users. While government, civil society organizations, and development partners in Honduras produce a wide variety of data, uneven access and low visibility are key barriers in putting this information to use in development decision-making.

Section 3.2.2.1
Access: Absent a culture of information sharing, data is often available, but not accessible

Government ministries or units that collect raw data do not always share it, or there are delays in putting these data into the public domain.72 A few government respondents acknowledged that there was no culture of sharing information, and that information from other ministries would be very useful in their work.73 While the government may release PDF reports, many of the underlying databases it maintains are classified, which impedes broader use of these data among the public.74 Despite the existence of high profile government transparency initiatives, several interviewees have actually observed a decrease in their access to government data in recent years, as information is hidden behind either paywalls or firewalls.75

There is also a perception that development partners do not always make public the results or outcomes of their projects. Several government and civil society group respondents reported that development partners sometimes bypass the central government to approach municipalities directly for information, or support closed databases that collect similar or duplicate data.76 This challenge was not limited to data on project inputs, but also included information on lessons learned and project outcomes, which development partners do not always make public.

Civil society organization participants expressed a need for more information on the results and impact of projects to enable learning. This was not specific to projects implemented by CSOs, but a perceived need in general. It was mentioned that when projects fail, the reaction is often to hide the data, which does not allow for other actors to learn from those mistakes. This is a missed opportunity for other parts of the government or non-governmental actors to access the same set of information and iterate together.

In the case of non-reimbursable funds, [development partners] don’t feel the commitment to report back to the government. If the donor wants to hide, they hide.
Government Respondent
Section 3.2.2.2
Familiarity: Even when data is accessible, it may not be visible enough to prospective users

People cannot use development data that they do not know exists. Certainly, this is true for proprietary databases and reports that are hidden from public view, but this can also affect open data portals if they are not proactively communicated and promoted amongst prospective user groups. Honduras’ aid information management system (PGC) is case in point: despite being open to the public, only a third of the government and development partner representatives interviewed (and even fewer CSOs) were even aware of its existence. 77

Duplication of efforts in development cooperation (and in data collection specifically) was cited as a concern across stakeholder groups. Government officials spoke of poor coordination with development partners, and between development partners and mayors of municipalities; development partner and civil society organization officials lamented poor coordination among development partners. Most respondents expressed a desire to know what others are doing, particularly in their sectors of interest, and the need for greater synchronization of efforts (and funding) among all these actors.

Strikingly, these same groups reported limited awareness of the aid information management system (PGC), which has publicly accessible project-level information that can be used to facilitate coordination within and between government agencies and development partners on development assistance activities.78 Unfortunately, the lack of visibility of open data sources like the PGC makes it difficult to stoke demand from domestic constituents, which dampens interest in addressing supply-side constraints to produce timely and comprehensive development data.

Section 3.2.3
Choice

The true value of development data is seen when citizens and officials take some action based upon this information to make recommendations, articulate preferences, or change their own behaviors. In the case of Honduras, government, development partner, and civil society representatives reported using at least one data source, and many indicated a desire for more data and analysis in their work. Government representatives primarily draw upon proprietary or open government data sources and studies published by large international organizations such as the World Bank and USAID. Development partners use a mix of data produced by the government and other development partners such as the United Nations and World Bank. Civil society groups rely the least on government sources and more on development partner sources and their own data collection. However, there remain consequential deterrents to evidence-informed decision-making.

Section 3.2.3.1
Credibility: Government-produced data is plentiful, but there is a trust deficit with prospective users of this information who doubt its veracity

As is often the case in low- and middle-income countries, government-produced data in Honduras appears to suffer from a trust deficit, which undercuts use. CSO representatives in particular raised concerns that mayors and other officials sometimes skew data to receive more funding (e.g., adjusting population figures) or under-report bad news to paint a rosier picture than reality. Interviewees raised examples such as: teachers abetting students to cheat to boost test scores, nurses not registering people with Chikungunya (a mosquito-borne illness) to lower the disease incidence rates, and the lack of sound population data making it difficult to calculate the percentage of children in school.

Section 3.2.3.2
Incentives: Data is only one part of the calculus, and leaders may still ignore evidence in favor of convenience or organizational imperatives

Even when government, development partner, and CSO leaders have access to credible and relevant development data, there are limits to what this information can do. Individual incentives, long-standing country strategies, and organizational imperatives often compel decisions to be made based on convenience or intuition, rather than evidence. The challenge of curbing aid fragmentation and improving coordination is particularly illustrative.

I don’t [put much] trust in government data. When I go to the field the reality is always different.
CSO Respondent

Development partner and government representatives frequently express the need for better coordination, but there is little indication that they use data to allocate resources and plan aid activities. For example, one respondent cited a sector roundtable (mesa sectorial) that collected information from development partners on the amount and location of investments, and identified areas of duplication. However, no real measures were taken to improve aid coordination in the sector. Development partners still fund contradictory activities in the same communities or duplicate efforts despite the available evidence.79

People want to use their common sense and not the data…They think that the data is telling them things that they already know, and so they don’t think it is important
Development Partner Respondent

Section 3.3
Recommendations: Where do we go from here?

Honduras has taken important steps towards creating the legal and institutional infrastructure for open data; however, many government databases are not yet public or have restricted access to only certain ministries.80 Substantial data gaps remain for government officials, civil society organization representatives, and development partners to fill and monitor progress against national development priorities and global goals. Even when data is available, there are other barriers to use. In this final section, we propose recommendations to close the gap between demand and supply for data to support resource allocation, project implementation, and monitoring and evaluation.

Section 3.3.1
Recommendation #1: The Government of Honduras should strengthen the capacity of the National Statistics Institute (INE) to publish surveys and censuses in line with domestic demand for more granular data on sector-specific outcomes

Historically, the INE has leaned heavily on external financing for its data collection activities. While development partners can play a supporting role, data and statistics should be a domestic priority. Specifically, the government should prioritize additional financing and technical support for the INE to be responsive to a growing demand for timely, disaggregated, and geo-referenced data on sector-specific development outcomes. Investing in more regular censuses and household surveys could be a good place to start. Central government data leaders should cultivate political buy-in from municipal counterparts to provide greater support and security for enumerators. In addition, the government should remove pay wall restrictions on INE datasets to ensure that this high-value information is freely available to all.

Section 3.3.2
Recommendation #2: The Government of Honduras should increase awareness of open data portals such as the public aid information management system (PGC) and streamline reporting for development partners to improve data quality and coverage

The fact that there is such limited public awareness of Honduras’ aid information management system (PGC) is a missed opportunity to demonstrate demand and incentivize government and development partner leaders to invest in opening up other data sources. The PGC’s host institution, the secretariat for external relations (SRECI), should make a concerted effort to publicize and increase awareness of the portal among other government ministries, development partners, and CSOs, perhaps through a series of trainings, tutorials and/or analytical products that show the value of the data.81

At the same time, the SRECI should alleviate concerns regarding timeliness, salience, and coverage raised by those familiar with the PGC. Streamlining reporting for development partners so that they do not need to report into multiple systems (e.g., line ministry databases and the PGC) could reduce the burden of keeping information up-to-date and complete. Meanwhile, the SRECI should coordinate with other line ministries and development partners to identify additional high-value information that could be included to increase the perceived utility of the portal, such as maps that combine aid project locations with sector-specific indicator data.

Section 3.3.3
Recommendation #3: The Government of Honduras, development partners, and civil society organizations should employ the results-based management system (SGPR) as a signature effort to cultivate a strong culture of information sharing by opening up data on the entire service delivery chain

At present, the government’s new results-based management system (SGPR) operates as a parallel system: access is limited and reporting occurs apart from the normal databases managed by line ministries. The Office of the President should appoint a task force of high-level champions in each ministry to facilitate interoperability between the PGC (aid data), SIAFI (budget data), and the SGPR (results data) to bring together data on the entire service delivery chain – from resourcing to results. This could also include other data produced by development partners and civil society groups that are seen as relevant and credible to augment official statistics.

The Government of Honduras should also make the SGPR data publicly available, while protecting any personally identifiable information of citizens and beneficiaries of development projects.82 Opening up this information could generate several positive ripple effects, such as: increased access and use of government datasets, reduced data collection and management costs, and the diffusion of new norms of information sharing.

Chapter 4

Data Use in an Oral Culture: Putting Development Data to Work in Timor-Leste

Authors

Alena Stern, Dina Abdel-Fattah, Taryn Davis, Paige Kirby, and Lauren Harrison

Seeking to put the country on a trajectory to achieve middle-income status by 2030, the Government of Timor-Leste established the 2011 Strategic Development Plan (SDP)83 and officially endorsed development principles codified in the New Deal for Engagement on Fragile States.84 Former Prime Minister Xanana Gusmāo also sought to build a “full chain of accountability” into the government, opening up data for the public to actively monitor the government’s financial management and progress against the SDP.85

Timor-Leste faces a reality that is becoming more common among developing countries: access to information is relatively high, but there is lower capacity to use this data, which hampers its impact. This study examines the prospects and limits for Timor-Leste to translate greater access to data on investments (e.g., domestic resources, foreign aid) and outcomes (e.g., project results, national statistics) into progress against the country’s goals. What data is available and how are decision-makers using this information to manage resources? What barriers must these leaders overcome to use this data effectively?

With a focus on the health and agriculture sectors,86 the research team put these questions to the test. We interviewed 77 leaders from three critical stakeholder groups: government agencies, civil society organizations (CSOs), and development partners. They offered their insights on the individuals, institutions, and incentives that impact the supply and use of development data in Timor-Leste.87

Section 4.1
Landscape: Who produces data, who uses it, and why?

Although government agencies, development partners, and CSOs serve distinct functions, they operate interdependently to support Timor-Leste’s development progress. In this section, we examine the decisions that these stakeholders routinely undertake in Timor-Leste and the types of development data they produce and use in their work.

Section 4.1.1
Government of Timor-Leste

The annual budgeting and planning process has significant bearing on the resources available for national development programs in Timor-Leste.88 The Ministry of Finance works closely with each ministry to prepare budget submissions based upon available resources and national priorities.89 Once Parliament votes to pass the budget, line ministries prepare their Annual Action Plans using multi-year sector-specific strategies to program activities.90 This strategy development process is often led by external consultants in coordination with the line ministries and may include a desk review of census and survey data, consulting past reports, as well as consultations with government, development partner, and civil society stakeholders.

Section 4.1.1.1
National budget planning and implementation is an important entry point for data on development resources and outcomes

The Ministry of Finance monitors budget implementation through an expenditure review analysis unit which tracks the status of projects based upon reports generated from its Performance Management System.91 Line ministries develop their own information management systems to collect project implementation and results data (e.g., the Ministry of Agriculture and Fisheries’ Suco-Level Monitoring System and the Ministry of Health’s Health Management Information System).

Line ministries also collect information on the sector-specific activities of development partners to support budgeting, inform planning, and monitor performance. In some cases, they employ their own reporting templates to obtain information via sector-specific working groups (e.g., agriculture and health).92 In addition, development partners report their activities to Timor Leste’s aid information management system, the public Aid Transparency Portal (ATP) managed by the Ministry of Finance’s Development Partnership Management Unit.93 The ATP replaced a previous Excel-based data collection method, and is the most comprehensive source of aid information in Timor-Leste. Table 1 provides a description of various sources of aid information available in Timor-Leste.

Ministry officials reported using data not only to plan, execute, and monitor their own programs, but also to inform their interactions with other parts of the government, foreign governments, and development partners. For example, Ministry of Finance respondents noted that they are requested to pull information from the Aid Transparency Portal to provide reference information for the Ministry of Foreign Affairs in advance of Policy Dialogue meetings with external partners that provide foreign assistance.

Table 1: Sources of Aid Information

Aid Transparency Portal (ATP):

The official aid information management system (AIMS) of the Government. Launched by the Ministry of Finance’s Development Partnership Management Unit in July 2011. Includes information on official development assistance, including: financing, project descriptions, and sub-national project locations (aidtransparency.gov.tl).

Sector-Specific Systems:

Systems created by line ministries to collect data on relevant development partner-funded programs. This information is often collected directly from development partners via sector working groups. The information collected is similar to that in the ATP.

Belun National Database:

Belun, a Timorese NGO, created a national database in 2014 with funding from the Civil Society Unit of the government that aims to capture information on NGO programming in Timor-Leste. It is distinct from the ATP in its focus on community-level actors and is often collected directly from these groups and in coordination with FONGTL, the national NGO forum in Timor-Leste (http://www.belun.tl/en/community-capacity-dev/national-database/).

Professional and Social Networks:

Interpersonal interactions are a key source of aid information. This may be formalized through sector or stakeholder working groups or formal consultations during project design. “Ad hoc” interactions and social networks, such as the East Timor Action Network (ETAN) email listserv and Facebook were also cited as useful sources of information.

Section 4.1.1.2
The government voluntarily discloses information via open data initiatives; however, these transparency gains are not durable absent a Freedom of Information law

Under the Gusmāo administration, Timor-Leste launched several open government portals to disclose information, including: the Budget Transparency Portal, the eProcurement Portal, the Aid Transparency Portal, and the Government Results Portal. It also established itself as a global leader in transparency: joining the Extractive Industries Transparency Initiative (EITI) in July 201094 and receiving a ranking of “comprehensive revenue transparency” in the 2010 Revenue Watch Index.95 These efforts on the part of the government were bolstered by investments from the World Bank, the IMF, and others to build the country’s national statistical capacity.96

However, Timor-Leste lacks a formal Freedom of Information (FOI) law and has only weak protections for the media (Freedom House, 2015).97 CSOs go through lengthy and difficult formal request processes or rely on their personal and professional networks to access data.98 Therefore, while the government has, laudably, chosen to make some data public via its Government Transparency Portal, accessing other high-value government data sources is difficult.

In the absence of clear guidance for what data should be made publicly available, many government officials default to keeping data private until given explicit permission to share. There is a strong “culture of deference and respect for hierarchy” that extends to data disclosure practices – senior-level government approval is often required to access or release information (Freedom House, 2015).99

Section 4.1.2
Development Partners

Timor-Leste generates 90% of its revenues from oil, making its growth extremely vulnerable to commodity price shocks (UNDP, 2012).100 Declining oil prices in recent years have set the stage for official development assistance (ODA) to play an increasingly important role in Timor-Leste’s development.101 This is particularly true of social sectors such as education, health, and agriculture that faced significant budget cuts in the country’s most recent budget.102 Development partners, therefore, make numerous decisions that impact the country’s development: influencing priorities, allocating official development assistance (ODA) resources, and designing discrete activities.

Section 4.1.2.1
Development partners are power users of development data, but strategic imperatives diminish the role of objective evidence in decision-making

Although development partners vary in their division of labor, typically several actors inform the decision-making process – each serving distinct, but complementary, roles. Headquarters may provide strategic guidance, typically in the form of broad goals to inform country strategies, or they may take a more “hands-on” approach to developing or approving strategies to be implemented by Country Offices at the national level.103 Within Country Offices, Program Officers oversee operational or technical areas of a development partner’s portfolio and manage the activities of implementing partners, while Technical/Support Offices serve crosscutting functions across projects (e.g., finance, monitoring and evaluation, etc.).

Development partners reported using data on development outcomes, past project performance, and the activities of other development actors to: formulate strategic plans, assess needs, design projects, and establish baseline outcome measures to monitor progress. Development partners often seek to leverage existing data, such as government statistics (e.g., census) or third party data (e.g., Demographic and Health Surveys or Asia Foundation data) for these purposes. However, many respondents reported that they were unable to find existing data that was fit-for-purpose. In these instances, development partners fund primary data collection efforts to be implemented internally or by an external party.

Yet, interest in data does not always translate seamlessly into decisions. Development partner decisions may be driven by strategic imperatives from headquarters rather than available evidence, reducing the motivation of development partner officials to use data. Since most development partners implement programs through contracted services, their implementing partners make consequential decisions about the design, implementation, and monitoring and evaluation (M&E) of projects. Data used by development partners is most likely to impact these decisions when written into the terms of reference or directly passed along to their implementing partners.104

Some development partner respondents are evaluating how to collect M&E data from implementing partners that can most effectively be used as a decisional input by development partner staff. For example, the Australian Department of Foreign Affairs and Trade is launching a new M&E program that will institute an integrated performance management system focused on increasing uptake of M&E data in decision-making.105

Section 4.1.2.2
Development partners are interested in using development cooperation data to curb fragmentation and improve targeting, but are frustrated by duplicate reporting systems

Given the number of development partners operating in Timor-Leste, coordination is a strong use case for official development assistance (ODA) information to avoid duplication of efforts. Most development partners expressed confidence in their ability to obtain a complete picture of relevant development cooperation activities during project design through extensive, in-person stakeholder consultations. However, others recognized the potential for a skewed picture that excludes smaller actors or those that weren’t available to attend a meeting during the consultation period.

In addition to personal networks, many development partners reported using Timor-Leste’s public Aid Transparency Portal (ATP) to identify the right individuals to speak with for stakeholder consultations and select project locations. Development partner respondents also raised other use cases for the ATP to inform more effective targeting of their assistance to districts with the greatest need and learn from experience to improve project design and avoid repeating mistakes. However, some development partners expressed frustration with the duplicate reporting requests they receive to report into the ATP and via the sector-specific schemes.

Subnational information, improving development partner coordination, and ensuring “no one is left behind” may go hand-in-hand. For example, when collaborating with the Ministry of Agriculture and Fisheries to design a new agriculture program, one development partner reported that they used the aid map in the ATP to identify potential gaps in coverage and discovered that most of the proposed project locations were in districts in the western part of the country, which was already receiving comparatively more agriculture funding than the east. Armed with this information, the development partner insisted that the Ministry of Agriculture and Fisheries include some eastern districts in the program.

Section 4.1.3
Civil Society

A broad spectrum of domestic and international civil society organizations (CSOs) inform and shape Timor-Leste’s development. These organizations produce a variety of development data and seek additional data produced by others as inputs to inform strategy development, project implementation or research and analysis.

Section 4.1.3.1
Civil society organizations may fund their own data collection efforts to fill perceived gaps

As CSOs implement projects and conduct primary research, they produce and collect data that can be of broader use to other development practitioners and policymakers. In some cases, CSOs self-fund data collection efforts to fill perceived data gaps, even when external resources are not available. For example, when confronted with a lack of tourism data, the Asia Foundation piloted an effort to compile this information by surveying individuals entering Timor-Leste at the airport, which facilitated a full-scale study.106 However, the self-funding model is only feasible for those organizations with the resources – both time and money – for the effort.

Section 4.1.3.1
Civil society organizations highly prize disaggregated and geo-referenced data at the sub-district level

CSOs reportedly consider several factors in determining whether to undertake new activities, including: alignment with organizational strategy, availability of funding, and their geographic presence. Funder priorities may influence the selection of target districts and activities, but CSOs typically determine the specific project locations (i.e., villages or communities). Therefore, it is perhaps unsurprising that CSO respondents expressed greater demand for spatially precise data on development indicators at the sub-district level than other respondents. Precise information on official development assistance-funded activities at the sub-district level is also important to CSOs that need to coordinate with other organizations working in the same geographic area or want to identify prospective funders of their work that are active in their sectors of expertise.107

Table 2 provides a recap of the main decisions made, data types used, and unmet data needs of each of the major stakeholder types included in our study.

Table 2. Data Use and Data Needs by Stakeholder Group in Timor-Leste

Decisions

Data Sources Used

Reported Data Source Needs

Central Government

Annual budget allocation, execution, and monitoring

Government Revenues (Central Bank)

Past Expenditures (Performance Budgeting System)

Government Priorities (Strategic Development Plan)

Foreign Aid (Aid Transparency Portal)

Integrated aid and budget data

More precise data on aid funding to government ministries

More detailed, timely, and complete reporting on aid activities by donors

More accessible historical budget data.

Line Ministry

Annual budget proposals and action plans;

Sector strategies;

Development partner coordination;

Program M&E

Foreign Aid (Aid Transparency Portal, development partner consultations, information collected directly from development partners via ministry systems)

Development Outcomes (National Directorate of Statistics)

Program M&E (line ministry information systems such as the Suco-Level Monitoring System and the Health Management Information System)

Government Priorities (Strategic Development Plan)

More precise data on aid funding to government ministries

More precise information on development programs and inputs (especially for research units)

Development Partners

Strategy development;

Program design;

Coordination with line ministries and development partners;

Program M&E

Development Outcomes (National Directorate of Statistics, Third-Party Sources such as DHS, World Bank, etc., and self-collected data)

Aid Activities (Aid Transparency Portal, data collected in working groups, in-person consultations)

Program M&E (self-collected)

Reliable subnational statistics disaggregated at the district level (e.g., unemployment)

Standardized indicators associated with the Strategic Development Plan

Implementing CSOs

Strategy development;

Program design;

Coordination with other institutional partners;

Program M&E

Development Outcomes (National Directorate of Statistics, Third-Party Sources such as DHS, World Bank, etc, self-collected data, stakeholder consultations)

Activities of Other International NGOs (INGOs) / Implementing Partners (IPs) (direct consultations, INGO working group)

Program M&E (self-collected)

Inclusion of CSOs in Aid Transparency Portal

More accurate, timely, and precise outcome data

More precise subnational data sources/ greater geospatial data coverage

Research CSOs

Research issue selection and design;

Research production

Statistical Data for Research Inputs (National Directorate of Statistics data, self-collected data, Third-Party Sources – in some cases provided directly by funder)

Stakeholder Needs (community consultations, surveys)

Government Budget Data (Ministry of Finance)

NGO/CSO consultative input during government budgeting process

Access to raw data in government transparency portal to conduct analysis

More precise subnational data sources/ greater geospatial data coverage

Section 4.2
Data Use: What are the binding constraints?

While specific use cases vary across stakeholder groups, respondents generally consider two key factors in selecting development data sources. First, they assess which data are most appropriate to answer their questions. Second, they consider the trade-offs: the costs of using a given data source, considering their own time constraints, skill levels, and organizational realities, versus the potential benefits.

In this section, we further explore the binding constraints to data use in Timor-Leste, as identified by interview participants, by adapting a theory of change previously developed by Custer et al. (2016) and presented in Chapter 2 as a common lens to view trends across the three country studies. This approach presents the causal logic of getting from data to impact (e.g., improved development outcomes) as the interaction of four C’s: content, channel, choice, and consequence. Given the scope of this study, we focus on the first three C’s as foundational building blocks.108

Section 4.2.1
Content

Government, development partner, and CSO respondents alike expressed that they face significant challenges in finding data that is both “fit-for-purpose” (appropriate to answer their questions) and for which they have the capacity to easily use. This prompts some development policymakers and practitioners to use the “next best” available data source. For others, it deters them from using data at all. This gap between demand and supply indicates a breakdown of some key assumptions regarding the accuracy and timeliness of development data produced in Timor-Leste, as well as its perceived salience and interoperability.

Section 4.2.1.1
Accuracy: Development data that is incomplete or out of date raises skepticism among prospective users regarding the reliability of available information sources

Accuracy of government data sources was frequently cited as a concern, exacerbated by substantial resource and capacity constraints. Sectoral data systems, such as Ministry of Agriculture and Fisheries’ Suco-Level Monitoring System (SLMS) and the Ministry of Health’s Health Management Information System (HMIS), were identified as particularly unreliable. Many noted that the local-level officials responsible for reporting information into these systems are under-resourced, under-trained, and under-motivated, yielding incomplete and inaccurate reporting.109

As a result, these government data systems can produce wildly inaccurate statistics. One respondent reported a 30-40% gap between immunization coverage reported in the HMIS versus an independent survey, while another suggested that the SLMS consistently overestimates agricultural yields. These challenges are exacerbated by the recent budget cuts to key social sectors, reducing the ability of line ministries to collect and use high quality data.110

Development partner respondents expressed concerns that the public aid information management system (ATP) does not provide accurate information on the subnational locations of their activities.111 One development partner specifically noted that they would be more likely to use the ATP data to design projects or coordinate with others if they had greater confidence in the quality assurance performed on the data and the resulting data quality.

I don’t trust the information on a website because I don’t know how recent or thorough it is.
CSO Respondent

Timeliness was also raised as a challenge for data sources, given the costs involved in fielding data collection efforts and the difficulty of securing sustainable funding. While the frequency of the government’s major surveys and censuses are in line with typical best practices, several respondents expressed concerns that these data are out of date.112 Similarly, many development partners cited lack of timeliness as the reason they prefer to go to their personal networks for up-to-date information on development cooperation activities, rather than use data maintained by the government.113 However, timeliness concerns extend beyond government data. A CSO respondent echoed this sentiment for third-party data sources in Timor-Leste, noting, “I don’t trust the information on a website because I don’t know how recent or thorough it is.”

Timeliness constraints compound accuracy concerns, as the government creates projections based on the 2010 census to extrapolate more recent statistics (e.g., population). Though such projections are common practice, respondents expressed concern that these projections may not be accurate due to shifting dynamics, and have concerns about the accuracy of the original survey, with one government representative referring to such statistics as “estimates on estimates.”

Section 4.2.1.2
Granularity: Despite a “wealth of available data,” there is still a persistent call for greater disaggregation and sector-specific statistics

Respondents were mixed on the issue of data coverage. Some take an optimistic view: citing the “wealth of data” available in Timor-Leste or the fact that the country is “over-studied.” Others were more pessimistic, especially in the agriculture sector. Several respondents noted the lack of available government data on agriculture and called for an increase in sector-specific statistics in agriculture and for the completion of the country’s first agriculture census planned for 2018. One government respondent lamented, “In agriculture, 99% of data is incorrect or incomplete.”

Beyond sectoral gaps, other respondents pointed to insufficient disaggregation in government data and advocated for access to raw budget and census data, as well as suco-level (village-level) disaggregation of government surveys. Central government officials expressed limited interest in the subnational location of development cooperation activities; however, many respondents indicated that this information might be of greater interest to local government officials or development partners.

In agriculture, 99% of data is incorrect or incomplete.
Government respondent

Section 4.2.2
Channel

Beyond content, creating an enabling environment for data use also requires thoughtful attention to the channel(s) used to disseminate this information and engage with users. Unfortunately, perceptions regarding access to, and capacity to use, development data in Timor-Leste appear to depend upon where you sit. Legislative and connectivity constraints, as well language barriers and low levels of data literacy still deter use among many groups.

Section 4.2.2.1
Access: Absent Freedom of Information legislation or broad-based Internet usage, access to government data depends upon one’s network114

The view that government information was difficult to access was not universally held. Typically, less well-connected junior staff reported greater difficulty obtaining data directly and instead turn to open fora (e.g., social media and listservs) to obtain information. This is problematic for data-driven decision-making, as it is often these junior staff that are responsible for preparing analysis for their senior leaders to review. Respondents in senior positions had a more sanguine view, noting that government data was comparatively more accessible in Timor-Leste than other countries in the region, and that they could easily gain access to government data via informal channels.

While personal and professional networks in Timor-Leste can facilitate information sharing, in some cases lack of awareness of available data sources inhibits uptake. For example, we found that many respondents were either completely unaware of the existence of Timor-Leste’s public Aid Transparency Portal (ATP) or did not realize that they could access the data.115 As a result, the ATP is largely seen as a government reporting tool rather than a useful data resource.

There is also a pronounced digital divide: only 1.1% of the population uses the Internet and those who do lament that it is slow and unreliable (World Bank, 2014).116 Twenty-nine percent of respondents cited slow or no Internet access as hindering their use of data.

For example, one line ministry was forced to develop a parallel system to collect data on development cooperation programs directly from development partners due to declining Internet connectivity that made it impossible to effectively use the online ATP.117

This also extends to those contributing data via web-based reporting interfaces, as well. Some development partners noted that it could take 1-2 hours to report data into the ATP, while others had to travel to use computers made available by the government at the Ministry of Finance. These challenges can cause some development partners to report data to the ATP late or not report at all, compounding the trust concerns discussed earlier.

Section 4.2.2.2
Capacity: Low levels of data literacy and numeracy, as well as language barriers, inhibit the effective use of development data in Timor-Leste

A weak educational system perpetuates low levels of human capacity in Timor-Leste (UNICEF, 2013).118 Few adults have had mathematics training and literacy and numeracy levels are low; however, fixing the education system is perceived to be a lesser priority relative to other goals (Anis, 2007; Marx and Pinhero, 2013). Meanwhile, the highest capacity individuals are often recruited to work for development partners who can offer much higher salaries, leaving a shortage of skilled labor which “adversely influences the effective functioning of the government” (UNDP, 2012).

As a result, the government’s capacity to use data for decision-making is limited, with many respondents indicating that long-form reports and dense statistics simply do not resonate. Some respondents noted that the government has limited capacity and resources to step up to expectations that it leads sector strategic planning, coordinate the efforts of development partners, or make evidence-informed policy changes. However, others cited improvements in Timor-Leste’s education system following independence, and expect that this will bring more highly educated individuals into government and increase data-driven decision-making in the future.

As in other countries, senior leaders and mid-level technocrats often disagree on their preferred forms to receive data. Senior leaders prefer data to be interpreted for them and reported in polished analysis products. Meanwhile, mid-level technocrats often prefer to conduct their own analysis using raw data. These distinct use cases are often difficult for any single data product or portal to accommodate. For example, senior decision-makers expressed frustration with the complexity of the public Aid Transparency Portal, preferring to navigate quickly to top-line numbers. In contrast, mid-level technocrats felt that navigating through charts, maps, and reports slowed their ability to access and export raw data.

Language is another critical barrier for many Timorese, with information frequently published in English or Portuguese rather than the far more commonly spoken Tetum. Indeed, only two of six major National Statistics Directorate surveys are publicly available in Tetum.119 Several respondents emphasized that in the oral culture of Timor-Leste, easily understandable data visualizations120 and in-person outreach are important ways to circumvent these language barriers and to convey complex information more simply.

Section 4.2.3
Choice

Catalyzing a data revolution for sustainable development in Timor-Leste requires not only better information, but also the willingness of policymakers and practitioners to put it to use in their planning, advocacy, and research. Yet, actively fostering demand, in the face of an “oral culture” and the propensity for respondents to source data from their personal and professional networks rather than official databases, presents a substantial challenge. Specifically, discussions with government, CSO, and development partner interview participants point to root issues of credibility and incentives that must be addressed before we see more robust use of development data in decision-making.

Section 4.2.3.1
Credibility: Many development partner and CSO respondents view government data sources as lacking credibility and field their own independent primary data collection efforts

When asked about their use of official government data sources, 48% of development partner respondents and 38% of CSO respondents said they avoid or limit use of government data due to a lack of trust in the data sources. This pervasive distrust appears to stem from three underlying factors: lack of timeliness, lack of accuracy, and lack of coverage. Questions about methodological rigor and accuracy in how data was collected, curated, and published were also raised as challenges to data use.

For example, concerns were raised about whether the public aid information management system (the Aid Transparency Portal, or ATP) provides a complete picture of development cooperation. CSO activities are likely underreported, since these organizations do not currently report their programs in the ATP. Development partners may report on those CSO activities they fund; however, this misses information on projects funded by private donors or development organizations outside of the country. Similarly, development partners noted that the ATP may not capture all aid modalities (e.g., technical assistance, bilateral support to multilaterals). One development partner respondent argued that because the Ministry of Finance has prioritized the collection of financial data on aid programs, this comes at the expense of fully reporting other pieces of information, such as detailed project descriptions or project results.

In light of this trust deficit, many development partners and CSOs field independent primary data collection efforts to supplement or circumvent their need to use government data. However, several respondents noted the high risk of “reinventing the wheel,” as these ad hoc data collection efforts are seldom coordinated with each other, compounding challenges of duplication, and inefficiency. Organizations that engage in data collection, or even different departments within the same agency, may have perverse incentives that mitigate against coordination, such as securing contracts or budgets for themselves.121 When duplicative efforts are identified, it is often too late in the process to make course corrections, given reluctance to share early-stage plans.

Lack of confidence in available data comes at a substantial cost: duplicative data collection has created a patchwork landscape of disconnected datasets that are not easily comparable due to varying methodologies and scopes, and yield conflicting results. As they sought to reconcile data sources to create a summary agriculture report, one government respondent reported turning up a seemingly never-ending set of conflicting surveys.

Section 4.2.3.2
Incentives: Powerful external and internal pressures dilute the influence of development data in making evidence-informed decisions

Even when issues of trust, access, and capacity are overcome, there are no guarantees that development data will be used. Headquarters’ directives, tied funding, and alignment with long-term strategies may undercut the ability of development partners and CSOs to make data-driven decisions in their programming. Path dependence can lead development partners and CSOs to choose project locations based upon networks and existing infrastructure, rather than targeting project locations based upon need-based calculations or a desire to reduce duplication of efforts. Discomfort with delivering “bad news” can undercut frank disclosure of monitoring and evaluation results and hamper the ability to make course corrections during implementation. Professional and political incentives may lead decision-makers to ignore evidence in favor of a more expedient decision.

Meanwhile, many respondents perceive that senior-level government officials are predisposed to distrust data and not inclined to use it in decision-making due to their interactions with experts peddling advice ill-suited to the context in the immediate aftermath of independence. To counteract this backlash, several respondents emphasized the importance of cultivating strong interpersonal relationships based upon trust with Government of Timor-Leste units to promote data and foster policy change. They noted that this “quiet approach” through collaborative face-to-face conversations has yielded much greater success than arms-length data sharing or a confrontational approach. However, these important personal relationships are regularly disrupted by the frequent restructuring within the government of Timor-Leste.122

Section 4.3
Recommendations: Where do we go from here?

If Timor-Leste is to make progress against global goals and national development priorities, data is essential to supporting more effective use of scarce resources and greater accountability for results. This section outlines recommendations to promote greater use of data and evidence to allocate resources, monitor progress, and evaluate results of development policies and programs in Timor-Leste.

Section 4.3.1
Recommendation #1: The Government of Timor-Leste should enact Freedom of Information legislation and streamline the process to request access to government data to increase availability

Almost 15 years post-independence, Timor-Leste still has a relatively weak policy and legislative environment to facilitate transparency and availability of development data. The failure to pass comprehensive Freedom of Information (FOI) legislation creates uncertainty within the government regarding which development data should be made publicly available. While the government continues to publish data through the National Statistics Directorate and the Timor-Leste Government Transparency Portal, gaining access to other data sources can be difficult without clear FOI policies. For this reason, Timor-Leste should make passing FOI legislation and other policies to increase data availability a priority.

Section 4.3.2
Recommendation #2: The Government of Timor-Leste should standardize indicators and align data efforts to measure progress against the Strategic Development Plan (2011-2030)

While the Timor-Leste Strategic Development Plan (SDP) seeks to align development efforts, in practice the breadth of objectives enables actors to justify nearly any program under its auspices. This in turn decreases the SDP’s practical utility to facilitate substantive coordination between the government, development partners, and CSOs toward common targets. Meanwhile, the lack of standardized results frameworks or indicator definitions has allowed a proliferation of independently collected data sources that use different definitions and provide conflicting or incomplete information. Focusing data collection efforts around the SDP could jump-start a virtuous cycle: a clear application for more accurate, timely development data that incentivizes producers to improve data quality and, in turn, bolsters confidence among data users.

Section 4.3.3
Recommendation #3: Increase timeliness of the Aid Transparency Portal data with streamlined reporting and improve accuracy by incorporating greater financial precision

Revealed demand among government, development partner, and CSO respondents for more information on development cooperation activities has not translated into broad-based use of Timor-Leste’s public aid management information system (ATP). Skepticism regarding the timeliness and accuracy of official databases has fueled a preference among interview participants to source information from personal connections rather than open data. Streamlining ATP reporting procedures – simplifying technical definitions and data fields – could enable development partners to provide more complete and timely information for fields where the demand is strongest. Updating the ATP fields to capture more granular financial information that specifies the share of on-budget funding that will be received by line ministries would increase confidence among line ministries in the accuracy of this data to support their budget planning.123

Section 4.3.4
Recommendation #4: Use working groups for forward-looking coordination in collecting and disseminating development data across the government, development partners, and CSOs

Sector- or stakeholder-based working groups are an important means of information exchange on development activities in Timor-Leste. Development partners, CSOs, and government representatives are known to host or participate in one or more such groups on a regular basis. These networks are underutilized venues for forward-looking coordination of data collection efforts and more systematic dissemination of data products. More thoroughly integrating CSOs into these working groups beyond a token civil society representative would further facilitate data sharing. The formation of a specialized “research working group” could replicate the successful working group model and break down sector silos to facilitate sharing of development data and learning from collective knowledge.

Section 4.3.5
Recommendation #5: The Government of Timor-Leste and development partners should invest in the capacity and compensation of data collectors at the point of service delivery

Demand for sector-specific data in Timor-Leste is outstripping the capacity of line ministries to provide information that is sufficiently accurate and timely to meet user needs. For line ministries, performance monitoring and associated data collection efforts are often undercut by weak political commitment, insufficient financial resources, and staff turnover. Since data quality begins at the point of collection, line ministries should: invest in training end-line data collectors (e.g., agricultural extension workers, health clinic staff) on proper data collection methods, provide feedback on the use of the data they produce, and provide merit-based compensation to retain top talent and motivate data collectors to improve data quality.

Section 4.3.6
Recommendation #6: Government data producers should intentionally mobilize internal data use champions and train officials to be savvy users of their products

Frequent restructuring within the government has unintentionally reduced its capacity to analyze, use, and share development data. To help overcome this profound skills and knowledge gap, producers of government data (e.g., the National Statistics Directorate, the Development Partnership Management Unit, and line ministry directorates) could take on a more expanded role in training mid-level technocrats with the skills to analyze raw data and senior officials with the skills to interpret and apply this analysis. Replicating models, such as the Asia Foundation’s Policy Leaders Group, which connects data enthusiasts within key institutions, or incorporating data analysis into government training programs, could incentivize greater data use.

Section 4.3.7
Recommendation #7: The Development Partnership Management Unit should conduct outreach to increase the visibility of the Aid Transparency Portal among development partners and CSOs, as well as train them to use the tool in their work

Many respondents were either completely unaware of the existence of the Aid Transparency Portal (ATP) or did not realize that they could access the data or use it as an input for analysis. We recommend that current outreach and training efforts, which focus primarily on the individuals and processes for reporting data into the ATP, be coupled with outreach and training for the mid-level technocrats and senior decision-makers at development partners and CSOs that are target users of the data. This is particularly critical for CSO users, who are essential to the reporting and use of sub-district location information, yet are the least likely to be aware of the ATP. Leveraging the sector and stakeholder working groups to disseminate information about the ATP can help mitigate the disruptive effects of frequent government restructuring. Several respondents called for greater leadership from the Development Partnership Management Unit to articulate not only how they are using ATP data, but also how other development actors can use the data in their own work.124

Section 4.3.8
Recommendation #8: To broaden dissemination, data producers should translate more information into Tetum and contextualize data with visuals or narrative

The preponderance of development data in Timor-Leste is released as raw data tables or long form reports which may fall on deaf ears in a predominantly oral culture. Moreover, most of this data – particularly that which is collected and produced by external partners – is only available in English or Portuguese, rather than the more commonly spoken Tetum. The reliance on online dissemination channels to communicate and share official development data sources is also likely insufficient in Timor-Leste, where the vast majority of people have no or low bandwidth. For government, development partner, and CSO producers looking to broaden the reach and use of their data products and analysis, they should consider greater investment in disseminating information in multiple languages125 and effectively combining raw data with visuals or narrative for context.

Chapter 5

Harnessing the Data Revolution to Fuel Senegal’s Emerging Development Strategy

Authors

Jacob Sims, Harsh Desai, Oussenyou Ngom, and Vanessa Sanchez

Buoyed by a strong track record of peaceful democratic transitions and sustained economic growth, Senegal has garnered a reputation as a bright spot for good governance in West Africa (USAID, 2016; Pham, 2016; Petesch, 2016; Ba, 2016).127 In 2015, the Government of Senegal (GoS) launched an ambitious national strategy, the “Emerging Senegal Plan” (ESP), to tackle extreme poverty and achieve more inclusive development over the next 20 years (Kireyev, 2014; AllAfrica, 2015).128 President Macky Sall has subsequently emphasized a greater role for data to guide sector priorities and budget allocations under the ESP, beginning in 2017.129

As Senegal moves to achieve its aspiration of more data-driven decision-making, from where is it starting? How are decisions to target and evaluate development projects currently made? What opportunities and obstacles exist for development data to be used in making these decisions? With a focus on the education and health sectors, this study draws upon the first-hand experiences of 49 leaders from government, civil society organizations (CSOs), and development partners.130 They offered their insights regarding the decision-making landscape, the role of data, and the barriers to effective use of development data in Senegal.131

Section 5.1
Landscape: Who produces data, who uses it, and why?

Government officials, development partners, and CSOs routinely make a variety of decisions that influence Senegal’s development trajectory: setting sector priorities, allocating resources, monitoring progress, and making course corrections. In this section, we examine how these actors reportedly use three categories of development data – national statistics, sector-specific indicator and infrastructure data, and information on development cooperation activities – in their decision-making. We specifically focus on education and health, the pilot sectors for President Sall’s data initiative.

Section 5.1.1
Government of Senegal

In Senegal, all centrally coordinated development activities are shaped by 10-year national- and sector-specific strategy documents developed with input from the relevant line ministries. Each fiscal year, the Ministry of Finance prepares an Annual Investment Plan (AIP) outlining budgets and investments to align the country’s spending with progress against the 10-year strategies. In preparing the AIP, the Division of Economic and Financial Cooperation (DCEF) dialogues with line ministries and the President to incorporate their financial commitments towards certain projects, programs, and priorities into the budget.132

Section 5.1.1.1
10-year sector-specific strategies substantially shape not only planning and budgeting processes, but also the government’s data collection priorities

In the education sector, for example, the Ministry of Education establishes priorities aligned with its 10-year sector strategy and collects indicator data to track progress against these objectives. Functional directorates implement policies and programs for students in their assigned age-range and collect school-level indicator data for monitoring and evaluation. Regional-level indicators on school, teacher, and student performance that the ministry needs to track progress against the national education strategy remain the most sought after pieces of information.

The Directorate of Planning and Education Reform (La Direction de la Planification et de la Réforme de l’Education, DPRE) aggregates age-range and school-level indicator data from various functional directorates, as well as standardizing and verifying data received from development partners and CSOs. The DPRE uses this data to produce an annual statistical report on progress in the education sector. The Ministry of Education uses the DPRE annual report to communicate with development partners and CSOs to ensure strategic alignment of their activities with Ministry of Education priorities and advocate for funding on specific initiatives.

The Ministry of Education’s National Institute for Research and Action for the Development of Education (INEADE) also uses this information to conduct in-depth evaluations on desired educational outcomes metrics across the sector. The Ministry of Education draws upon these INEADE evaluations, the DPRE annual report, as well as other data from the National Institute of Statistics (NIS) to provide rationale for projects, monitor progress, and determine resource allocation for schools across the country. This, in turn, shapes the Ministry of Education’s annual strategy (Politique Sectorielle), which outlines its annual priorities and funding requests. Table 1 visualizes how government units work together to produce and use data to shape Senegal’s education policies.

In the health sector, Senegal’s national health strategy is primarily shaped by a steering committee comprising heads of various directorates within the Ministry of Health (MoH), the donor working group for the health sector, and the Council of Non-Government Development Support Organizations (Conseil Des Organisations Non Gouvernementales d’Appui Au Développement, CONGAD).133 Each year, these actors assess the strengths and weaknesses of the Ministry of Health’s strategy and make adjustments to better align performance with targets. These adjustments are reflected in the Ministry of Health’s annual sector strategy and inform coordination efforts.

Table 1: Decision-Making Responsibilities within Senegal’s Ministry of Education

Functional Directorates

Directorate of Planning and Education Reform (DPRE)

National Institute for Research and Action for the Development of Education (INEADE)

Pre-Primary

Primary

Secondary

Tertiary

Collects data from functional directorates

Produces annual statistical report

Evaluates performance of education sector

Informs ministry’s funding priorities

Section 5.1.1.2
Sector-specific indicator and infrastructure data collected by the national statistics agency and line ministries are highly prized for decision-making

Sector-specific information on development indicators and infrastructure are highly prized and utilized in planning projects and evaluating progress in Senegal. In the context of this study, indicator data refers to the current status quo in a sector and given geographical area. For example, in the health sector, this might include: child mortality rates, the number vaccinated against polio, and the percentage with access to health care. Infrastructure data refers to the landscape of physical inputs designed to improve upon a given indicator within an area. For example, this might include: the number of clinics and hospitals by district, or the supply of medicines, doctors, and equipment by hospital.

Line ministries collect much of this sector-specific information. For example, the District Health Information System (DHIS) produced by the Ministry of Health is the backbone for data-driven decision-making in the health sector. It serves as the one-stop shop for providing discrete, relevant, and timely data on health indicators and infrastructure at all levels. Data in the DHIS is collected at each clinic or hospital, entered at the district level, and verified at the district and regional levels to ensure accuracy and timeliness.

The Ministry of Health produces an annual statistical report using sub-sectoral and geo-referenced health indicators to monitor the state of health in the country. Government officials also reportedly use geo-referenced health infrastructure data, alongside other data sources, to inform resource allocation decisions. Despite this interest in hyper-local data on health infrastructure, health officials did not strongly express a desire for other types of aggregate or subnational data (e.g., external or domestic financing for health programs).

In addition to line ministries, Senegal’s National Agency for Statistics and Demography (ANSD) is another important source of sector-specific indicator and infrastructure data. Under the auspices of Senegal’s five-year plan for statistical data collection and analysis activities — the National Strategic System — much of ANSD’s data is compiled via surveys fielded at various intervals from annually to once a decade.

The majority of ANSD surveys are administered via paper or field-based interviews that are recorded and later aggregated and analyzed. However, the ANSD is currently experimenting with a new method of survey data collection through the Listening to Senegal project, funded by the World Bank. In this pilot, the ANSD provides families with cell phones for their personal use and periodically calls these beneficiaries to ask targeted questions to collect information for their surveys. The ANSD makes recommendations based upon analysis of these surveys, such as advising government decision-makers on areas of greatest need.

Section 5.1.1.3
Development cooperation data may be an underutilized information resource among government officials who use it primarily to report to development partners

While official development assistance (ODA) is an important source of financing for activities in Senegal’s education and health sectors, the primary users of information on the distribution of these dollars and project results appear to be development partners, rather than the Government of Senegal. Government use of development cooperation data is limited to the Ministry of Finance’s production of its annual Official Development Assistance (ODA) report. While development partners acknowledge that the ODA report is useful as a top-line overview of development cooperation activities in Senegal, this information is not widely utilized by line ministries for sector-performance monitoring.

Ministry of Finance officials typically access data on ODA programs either through their professional networks or via Senegal’s Aid Management Platform (AMP). Hosted by the Division of Economic and Financial Cooperation (DCEF), the AMP is an aid information management system that tracks foreign assistance and public investment projects from the national budget.134 Two part-time DCEF employees currently maintain the AMP and request inputs from line ministries and development partners via an Excel template.

The Aid Management Platform templates contain “activity reports” listing all necessary fields and are populated using internal project documentation. These excel templates are not technically challenging to complete, but they do take time to fill out and the political incentive or mandate for doing so remains unclear absent high-level political will or demonstrated use cases. As a result, the platform is often incomplete (e.g., missing projects and fields) and out-of-date – as of late 2016, data on activities from 2014 had not yet been published – which limits its utility to a rough assessment of the aggregate contributions from each development partner in a given sector, as opposed to reliable estimates and disaggregated analysis.

Delays in publicizing data, lack of disaggregation, and low visibility of the platform beyond the Division of Economic and Financial Cooperation undercut its potential to play a broader role in helping line ministries, CSOs, and development partners to visualize and manage investments to achieve the Emerging Senegal Plan (ESP). Even though the platform may have most of the functionality needed to contribute in this way, the Ministry of Finance is undergoing a separate review of its technical infrastructure.

Section 5.1.2
Development Partners

Education and health represent two of the largest sectors in Senegal receiving financial assistance from external development partners. As such, individual development partners and donor working groups play important roles, such as advising on the development of sector-specific strategies, funding discrete projects, and providing technical assistance to assess project impact. France, the United States, and Belgium are substantial contributors to health sector financing.135 In education, France, Canada, and the World Bank are prominent development partners.136

Currently led by the United Nations and Spanish government,137 the G-50 is the primary donor coordination mechanism in Senegal and comprises the 50 largest multilateral and bilateral development partners and international NGOs in the country.138 The G-50 seeks to increase the effectiveness of its development cooperation through two coordination mechanisms. The G-12 group of the 12 largest development partners139 coordinates with the Ministry of Finance on financial disbursements and liaises with development partners not formally embedded within the G-50, such as China. The G-12 also liaises with ten sector-specific working groups of development partners and international NGOs that coordinate development efforts with line ministries to advance national development strategies and development partner priorities.140

Section 5.1.2.4
Development partners are the primary users of development cooperation information, but they often bypass official systems in favor of informal networks

Development partners utilize a variety of data and evidence, including information on the activities of other donors, throughout the project cycle, conducting feasibility studies to plan and site new projects, as well as impact assessments at project closure. Detailed project documentation (e.g., scope of work, findings, baseline/endline studies) is of high interest to development partners seeking deeper insights on project design, results, and lessons learned. Development partners also expressed strong demand for disaggregated and geo-referenced information on official development assistance-financed (ODA) projects to determine where to site projects and avoid duplication of efforts.

Interestingly, in sourcing this development cooperation information, many development partners reportedly bypass official databases, such as Senegal’s aid information management system maintained by the Government of Senegal, in favor of obtaining ODA program data informally via sector working group relationships. This might involve development partners meeting with other organizations known to have run similar projects in the same region. Alternatively, development partners primarily use their own internal monitoring data and that of other development partners with whom they work closely.

While development partners regularly communicate with their government counterparts via several mechanisms, government interviewees noted that development partners do not always share data in a timely manner. Meanwhile, several development partner interviewees expressed the desire to see the Senegalese government take a more active leadership role to ensure the effective coordination of development financing and technical assistance. Nonetheless, the donor community in Senegal is divided on the value of investing in a more systematic system of information sharing to replace the status quo.

Section 5.2
Data Use: What aer the binding constraints?

Government officials and development partners interact frequently to formulate priorities, monitor and evaluate progress, and revise strategies to advance the country’s development goals. Data and analysis were noted as essential to shaping allocation decisions and monitoring sector performance. Yet, to what extent does the rhetoric of data use in Senegal match up with actual practice and is the data currently available seen as useful to support decision-making?

In this section, we discuss the binding constraints to data use in Senegal by adapting a theory of change previously developed by Custer et al. (2016) and presented in Chapter 2 as a common lens to view trends across the three country studies. This approach presents the causal logic of getting from data to impact (e.g., improved development outcomes) as the interaction of four C’s: content, channel, choice, and consequence.141 Given the scope of this study, we focus on the first three C’s as foundational building blocks.

Section 5.2.1
Content

An essential precursor to robust use of development data is a ready supply of information (i.e., content) that is viewed as important and reliable for prospective users in their daily work. While the government, development partners, and CSOs produce vast amounts of development data in Senegal, interview participants raised concerns regarding the salience and accuracy of this information, as well as persistent capacity gaps for sustainable data production at the local level.

Section 5.2.1.1
Salience: There is a mismatch between the types of development cooperation data available and the information that is most in demand

Development partners view disaggregated information on development projects as useful to help them pinpoint the right actors to speak with for advice when planning to work in a specific sector or region. However, they were less certain regarding the usefulness of the type of information currently available in Senegal’s aid information management system (AMP). Rather than data on official development assistance (ODA) project locations, outputs, and finances – information published by the AMP – development partners instead expressed interest in qualitative information (e.g., evaluations and lessons learned), sector-specific indicator data disaggregated by geography, and data on crosscutting issues such as gender parity, climate change, and governance.

Section 5.2.1.2
Accuracy: Cumbersome data collection and reporting practices increase the risk of missing data and publication delays

Despite the government’s efforts to modernize the national statistical system in the past decade (Mauru, 2010), much of Senegal’s development data is still collected using antiquated methods. Primary data collection at the school or clinic level is often conducted using pen and paper due to limited Internet connectivity, computer access, and data management skills. This locally collected data is then sent via mail to be processed and digitized for inclusion in a digital database at the district or national level. Such a manual system compounds problems of timeliness, incomplete coverage, and poor disaggregation of data that, in turn, impedes effective use of that data for decision-making.

However, even when modern data collection methods are employed, cumbersome reporting practices undercut the reliability of official data sources such as Senegal’s aid information management system (AMP). Government officials and development partners alike view the AMP as less reliable, and a “second best” source of information to line ministry-curated databases or just-in-time data transfers between development partners.

Specifically, respondents voiced frustration with the AMP due to: publication delays (e.g., 2013 data on development projects was released two years behind schedule), missing fields, lack of standardization across donors, and duplicate reporting requirements (e.g., development partners being asked to report to both line ministries and the Division of Economic and Financial Cooperation (DCEF) for inclusion in the AMP).

Section 5.2.1.3
Capacity: Greater investment is still needed to professionalize Senegal’s national data and statistics infrastructure, particularly at the local level

Under the government’s leadership, Senegal has made important strides in increasing its technical capacity to produce high-quality statistical data, particularly with regard to poverty and education. However, substantial barriers remain for the government, development partners, and CSOs to effectively collect and use a broader suite of development data for decision-making in Senegal.

Funding and bandwidth constraints impinge upon the National Agency of Statistics and Demography’s (ANSD) ability to conduct timely national-level surveys. Implementation of the United Nation’s Post-2015 Development Agenda is likely to exacerbate this capacity gap, as there are greater demands placed upon the ANSD to assist with data collection to monitor and report on progress against the Sustainable Development Goals.

These technical and financial constraints also extend to line ministries. The Ministry of Education and the Ministry of Health struggle to recruit qualified statisticians in light of higher salaries available in the private sector. Meanwhile, within the Ministry of Finance, the lack of full time staff dedicated to maintaining the aid information management system hampers the DCEF’s ability to update the platform and ensure the integrity of the data, such as through conducting quality assurance checks against standardized reporting conventions.

Several interviewees from development partners and the central government also highlighted the need to improve data management capacity at the local level,142 including greater standardization in the way in which data is collected, curated, and aggregated. Data auditing functions are minimal, which increases the risk of data being manipulated to suit political needs or for data processing errors to create inaccuracies in the data. Another suggestion was to bolster funding for CSOs to assist the government to collect and verify hyper-local development data.

Section 5.2.2
Channel

How does development data actually get into the hands of prospective users in Senegal? The dissemination channel(s) for information can positively or negatively shape the chances that official and unofficial data sources are put to good use. Government, development partner, and CSO interviewees acknowledged difficulties in the ease of accessing government-produced data. CSOs in particular may be more reliant on publicly available information than their development partner and government counterparts.

Section 5.2.2.1
Access: Data that is “out of sight” is also “out of mind” for prospective users

At present, data access relies upon the willingness of government agencies to voluntarily disclose information – something that they are not always incentivized to do. The absence of Freedom of Information legislation or executive branch transparency initiatives inhibits efforts to broaden the user base through increasing accessibility of these data. With official data sources locked behind pay walls and firewalls, the Government of Senegal is missing an opportunity to encourage data-driven decision-making within and outside of the government to advance the country’s national development strategy.

Civil society groups, for example, cannot readily access the National Agency for Statistics and Demography’s (ANSD) treasure trove of national statistical data. Even line ministries within the government have limited access to data disaggregated by location and demographics. Absent a high-level champion, Senegal’s aid information management system (AMP) – housed in a directorate-level agency three bureaucratic levels removed from the Prime Minister – was never publicly launched, despite political commitments to do so.143

Section 5.2.2.2
Networks: Satisfaction with sourcing development information via informal personal and professional networks depends upon who one knows

Development cooperation data makes for a poignant case study on norms and preferences of information sharing. While Senegal’s aid information management system (AMP) is not fully open to the public, development partners and government officials can still access the data. However, seven years after its initial implementation, many of these stakeholders did not even know that the AMP existed. Among those that were aware of the AMP, they were divided on the appeal of sourcing development cooperation data from a database versus seeking this information from their network connections.

In contrast, CSOs articulated the strongest demand for official development assistance (ODA) program data of any stakeholder group. This stakeholder group seldom is able to leverage the informal networks that development partners or government officials use to share information. Thus, CSOs would arguably have the most to gain from a publicly available, comprehensive, and up-to-date aid information management system. Several CSOs mentioned the value of this data to conduct feasibility studies to inform the siting of projects and resources. Others voiced an interest in producing independent analysis on development project investments and results for greater accountability.

Section 5.2.3
Choice

If Senegal is to realize President Sall’s vision of data-driven decision-making to advance its national development strategy, the data itself is only one part of the equation. Government, development partner, and civil society actors must be willing to use this information to allocate resources, monitor progress, evaluate results, and make course corrections. However, as interview participants noted, making evidence-based decisions is more difficult in practice than in principle, especially in the face of organizational imperatives and political realities.

Section 5.2.3.1
Incentives: Political factors often mitigate the ability of data to drive development decisions

Based on our interviews, it is clear that data is more commonly used at some parts of the decision-making process than others. Government officials and development partners report seldom using development data to set budgets and strategies. In this respect, development partner strategic imperatives and governmental politics appear to be more consequential than data in determining project placements or facilitating donor coordination. However, government officials and development partners are more likely to use data to plan and site new projects (as part of feasibility studies) or to assess results at project closure. This may imply that the distribution of resources is a far more politically contested exercise than project design and evaluation.144

Yet, even the use of statistical data can be derailed by politics. National Agency for Statistics and Demography (ANSD) analysis and resource allocation recommendations based upon census data are often outweighed by political calculations. The government’s aspiration to move towards making a greater number of decisions on data and evidence through its forthcoming National Statistical System may substantially alter this status quo; however, this will take not only greater data production capacity, but also a cultural shift in giving greater voice to data and evidence in critical resourcing decisions. Additionally, it was noted that resource allocation decisions are often made on a political basis, so the Government of Senegal or development partners do not always follow recommendations made by the ANSD.

Section 5.3
Recommendations: Where do we go from here?

In this section, we highlight three recommendations to improve the relevance, quality, and uptake of data in Senegal’s aid information management system (AMP) to advance the country’s national development strategy. Our assumption in making these recommendations is that understanding development cooperation investments and results is an important piece of the puzzle to inform resource allocation decisions, shape sector strategies, and monitor progress. In this respect, the recommendations, while focused on the AMP, have wider relevance as a case study in how to optimize an official data source for maximum uptake.

Section 5.3.1
Recommendation #1: The Government of Senegal should position the aid information management system (AMP) as a tool to support implementation and monitoring of the Emerging Senegal Plan (ESP)

Although not currently used in Senegal, the AMP has the functionality to track indicator data relevant to the ESP alongside investment information to inform the targeting, siting, and assessment of development projects. The Ministry of Finance should employ this unused capability to support implementation of the ESP and proactively position the AMP with line ministries as a valuable tool to allocate resources and monitor progress in alignment with the national development strategy.

A strategic repositioning of the AMP could simultaneously spark interest in, and appreciation for, the value of this official data source among a broader coalition of users and a more diverse set of use cases. However, successfully mobilizing the necessary cooperation between and within agencies would require a clear signal from government senior leadership and a dedicated high-level champion. As a next step, the Ministry of Finance could pilot the use of the AMP to track both investment and indicator data for the Emerging Senegal Plan (ESP) initially in the education and health sectors (the focus of President Sall’s data initiative), as a proof of concept for further review.

Section 5.3.2
Recommendation #2: The Government of Senegal should expand technical resources and training to improve AMP data quality, coverage, and timeliness

Increasing the capacity of government officials and development partners to effectively curate, vet, and update Senegal’s AMP is critical to remedying persistent data quality challenges and increasing the credibility of the platform with users. In keeping with the fact that the AMP is owned and managed by the Senegalese government,145 the Ministry of Finance should dedicate a recurring portion of its annual budget to hire one or more full-time staff member(s) to ensure that there is sufficient technical bandwidth within the Division of Economic and Financial Cooperation (DCEF) to sustainably maintain the platform.146

Numerous government and development partner respondents reported that uploading data to the AMP was confusing and difficult, contributing to delays in publishing timely and complete data. The Ministry of Finance should host biennial refresher training workshops to increase familiarity of development partner and line ministry staff with the AMP reporting fields. In addition, the Ministry of Finance could reduce the time it takes for development partner to update AMP data by allowing development partners to either import the data themselves with host ministry validation or provide Excel data for the Ministry of Finance to input. Decreasing the number and complexity of the required data fields could further ease the reporting burden for development partners. Since the data in the AMP will only be as good as the inputs provided, it is prudent for the government to do all it can to increase the timeliness and compliance of development partner reporting.

Section 5.3.3
Recommendation #3: The Government of Senegal should broaden awareness of the AMP with a public launch campaign and crowd in feedback from prospective users

At present, the Senegalese public has little knowledge of the country’s aid management information system (AMP), and only a limited cadre of development partner and government officials have access to the data. This is a missed opportunity for the government to demonstrate its commitment to data-driven decision-making and engage the Senegalese public in monitoring progress against the Emerging Senegal Plan (ESP) through opening up its development cooperation data. As part of President Sall’s signature data initiative, the Ministry of Finance should make a renewed commitment to a set timeframe to publicly launch the AMP or, at minimum, conduct a ‘soft release’ of the data into the public domain.

In addition to a public launch event, the Ministry of Finance could partner with line ministries and sector working groups to design and implement an awareness campaign targeted towards prospective power users of Senegal’s development cooperation data (e.g., data journalists, development partner staff, civil society leaders, local government representatives). The campaign could utilize media coverage, stakeholder consultations, launch events, and blogs, etc. A particular focus should be placed on demonstrating use cases that will resonate with stakeholders and soliciting their feedback on how the AMP could be further improved for their specific purposes.

Chapter 6

Conclusion: The Next Generation of Catalytic Data Investments

Author

Samantha Custer

In writing this Avoiding Data Graveyards report, the authors set out to understand how leaders make decisions about development priorities, projects, and progress in three countries and the role of data and evidence in those choices.

In Chapters 3-5, we illuminated the development data landscape of each country in detail and presented context-specific recommendations to better align supply and demand. In Chapter 2, we synthesized broader insights on the binding constraints that prevent policymakers and practitioners from using information as they allocate resources, site projects, and evaluate the results of development projects and policies.

This concluding chapter turns from reflecting on the past to navigating the future. What do insights on development data use (or disuse) in Honduras, Senegal, and Timor-Leste mean for the data revolution for sustainable development writ large? As governments and organizations undertake the massive exercise of strengthening national statistics systems, how do we ensure that this greater capacity for data production does not merely result in data graveyards – places where well-intentioned and meticulously collected information goes to die? We propose operating principles for consideration by those that fund, produce, and consume development data to overcome binding constraints, both technical and political, and get to data use.

Section 6.1
Operating Principles: Where do we go from here?

The ambition of the 2030 Agenda and the data infrastructure necessary to support action and monitoring of the Sustainable Development Goals (SDGs) is daunting. By necessity, governments and organizations will make cost-benefit calculations to prioritize and sequence their production and use of development data in light of budget and technical constraints.147 In this section, we draw upon the three country studies to identify a series of operating principles for national leaders and development partners seeking to prioritize catalytic data investments that are responsive to demand across public, private, and civil society actors. We organize these principles at each link in the theory of change to get from data production to use: content, channel, and choice.

Section 6.1.1
Content

Building the capacity of national statistics systems in the post-2015 era requires more than investing in data production alone. If prospective users do not deem existing development data to be fit-for-purpose, there is a high risk that this information will lie fallow.

Section 6.1.1.1
Principle #1: Producers should proactively engage citizens, companies, and civil society organizations as co-creators and validators of development data

Funders are rightly interested in strengthening domestic capacity to produce data. Advocates are motivated to push for more of this information to be open to the public. Governments and organizations may rush in to supply data in order to be responsive to citizens, national plans, or international commitments, but ultimately miss the mark if they fail to address concerns regarding granularity, accuracy, and gaps in coverage that undercut the use of that information.

The response to this quandary, Custer et al. (2012) argue, often pits two philosophies of data production against each other: “cathedral builders” versus “vendors at the bazaar”.148 In other words, should producers place the emphasis on taking time to perfect their data prior to release (the cathedral builders) or be quick to show their wares (the bazaar vendors), with the view that it is the responsibility of the “buyer” or consumer to beware of potential data inaccuracies?149

While data producers understandably face trade-offs in timeliness versus accuracy, the wisdom of the crowd, when harnessed properly, could enhance the cathedral. Beyond instituting a minimum level of quality assurance standards and procedures, the sole burden of verifying the accuracy of data need not fall on the producer alone.150 Government agencies, for example, could proactively enlist the support of citizens, civil society groups, and development partners to help catch errors in official data sources. For example, Custer et al. (2012) suggest crowdsourcing information from community members to remedy incorrect locations for local clinics and schools in Kenya.

There may also be an opportunity to crowdsource data from unofficial sources to augment or supplement the official record.151 Development partners produce data that is seen as reliable and credible by other stakeholders. Civil society organizations are a relatively untapped resource for project-level data about the communities in which they work. Individual citizens are well-positioned to speak to the quality of public services or local prices for basic goods and services. For instance, Custer et al. (2016) highlight a project of the Philippines Ministry of Finance, the World Bank, and the technology company Premise to mobilize Filipinos to report on local cigarette prices and monitor tax enforcement that generates revenues for public healthcare.

Closer partnerships between official and unofficial data producers could increase public awareness of government data sources and expand their coverage and completeness. Of course, this would require government producers to put a spotlight on potential limitations, gaps, and deficiencies in the data. Since the world of data is not immune to politics, this recommendation is likely most feasible in contexts where high-level government champions are able to create an authorizing environment for agencies to openly share data and collaborate with end users. Without this political cover, inertia will be difficult to overcome.

Section 6.1.1.2
Principle #2: Funders of data initiatives should align incentives of producers to favor integration and interoperability

The anticipated cost of producing data to support global goals and national priorities is substantial, and leaders in low- and middle-income countries already face a continual challenge to free up sufficient domestic resources to support their national statistics organizations and the data needs of line ministries.152 To get the most capacity for limited dollars, governments and their development partners should seize the opportunity to tie domestic and external financing for statistics and data with the expectation that producers demonstrate greater integration, interoperability, and openness in their data collection, management, and publication practices.

Funders of data and statistics initiatives should make it more difficult for government agencies, development partners, and civil society organizations to undertake duplicative data collection activities or maintain siloed data management systems. They should encourage both international and domestic actors to share information along the whole service delivery chain from upstream resourcing and infrastructure to downstream indicators and outcomes. Funders should also support the leadership of national statistics organizations to identify standardized indicators and definitions to guide measurements of development progress that are cross-walked to both country-level national development strategies and international frameworks like the Sustainable Development Goals (SDGs).

Devarajan (2011) gives a practical example of what creating the right incentives could look like, arguing that the World Bank and other development partners should ensure that they are funding data and statistics that are well aligned with national statistical development strategies. By implication, this requires funders to take the long view, eschewing one-off data extraction exercises that would allow them to quickly get data for their own purposes, and instead investing with an eye towards sustainability so that domestic actors have the capacity to produce and share data that meet their needs.

Badiee et al. (2016) propose the formation of data compacts that governments and development partners would use as a guiding framework to pool their resources, in response to a common set of statistical capacity priorities in line with a country’s national statistics development strategy. Meanwhile, in their assessment of African statistics systems, Sandefur and Glassman (2014) encourage domestic and external funders of data initiatives to explicitly invest in “getting administrative and survey data to speak to each other” to validate information and provide a more holistic (and accurate) picture of development progress (p. 24).

Section 6.1.1.3
Principle #3: Funders should invest in regular collection and proactive disclosure of geo-referenced survey and census data as a demand-driven priority

Domestic demand for disaggregated data frequently outstrips the capacity of national statistics organizations (NSOs) to supply this information. Funders wanting to make demand-driven investments in data and statistics should take note of this capacity gap and invest in the capacity for NSOs and other actors to conduct censuses and surveys in sectors such as education, health, and agriculture with greater frequency. Badiee et al. (2016) recommend an expanded program of censuses and surveys, which provides a good starting point for funders and official data producers to determine the optimal frequency of these data collection exercises (see Table 1).

Yet, these investments will generate real dividends only if producers geo-reference and disaggregate census and survey data by key demographic attributes (e.g., age, sex, disability status). As Sandefur and Glassman (2014) note, when household surveys provide only national estimates, they offer “little guidance to domestic policymakers [seeking to] allocate resources and attention between subnational units” (p. 6). Civil society organizations and local governments have the most to gain from access to precise subnational data, but often have the greatest difficulty in accessing this information.

Similarly, policymakers are hard pressed to ensure that no one is left behind if they are unable to monitor differential access to resources, services, and progress for vulnerable groups. As Badiee et al. (2016) argue, the implication of the Sustainable Development Goals (SDGs) is the need to disaggregate data by “the characteristics of people and their locations”, while still enabling aggregation and comparability across space and time (p. 13).

Funders and official data producers must also expand coverage of existing censuses and household surveys to include under-represented groups. According to Stuart et al. (2015), we have a “missing millions” problem: an estimated 350 million people worldwide are not currently covered by household surveys and, for all practical purposes, are invisible to those seeking to allocate resources and monitor progress against the SDGs.

Beyond the desire to be inclusive, ensuring that these “missing millions” are counted is imperative for three reasons: to accurately estimate whether the lives of the poorest of the poor are improving, to effectively target resources to those in greatest need, and to enable vulnerable groups to have a credible evidence base with which to advocate for better services and support.

Table 1: Ten Year Program of Censuses and Surveys

Type of Survey or Census

Proposed Frequency

Source: Badiee et al. (2016, p.14). State of Development Data Funding. Retrieved from: http://opendatawatch.com/wp-content/uploads/2016/09/development-data-funding-2016.pdf

Population Census

Once every 10 years

DHS-MICS type surveys

Every 2 or 3 years

Living standards or household budget surveys

At least once every 5 years

Labor force surveys

Annually

Business establishment surveys

Annually

Agricultural surveys

Annually, depending on need

Time use surveys

Annually

Literacy / numeracy surveys

Once every 5 years

Victimization or related surveys

At least once every 5 years

Other surveys for national needs

At least once every 5 years

Section 6.1.2
Channel

The best quality development data will fall flat if prospective users cannot easily find, access, and use this information in their work. Funders and producers of data and statistics must proactively overcome persistent information asymmetries, both intentional and inadvertent, which inhibit the ability of decision-makers to take action (Custer et al., 2016).

Section 6.1.2.1
Principle #4: Producers should make data “open by default”,153 while funders and transparency advocates should work with them to make access sustainable

If funders want to get a better return on their investment and producers hope to increase uptake, both groups must proactively work together to broaden the reach of development data beyond their own organizations or agencies. They may find willing allies among open data hacktivists, open government reformers, and transparency activists.

There is growing recognition that national statistics should be open and that transparency initiatives should create norms and standards to ensure that people can easily access information they value. Data producers, both inside and outside of government, could benefit greatly from supporting broader transparency initiatives as a megaphone to promote visibility of, and access to, their data products. In return, transparency advocates may gain additional credibility and support for reforms if they are able to obtain the endorsement of agencies and organizations that are willing to open up their valuable development data.

Yet funders may have to break out of their technical or operational silos to exploit these opportunities to create an enabling environment for development data use. Funders of data and statistics capacity could make their investments contingent on producer willingness to adhere to open data principles and ensure sustainable access to that information through supporting FOI legislation.154 Meanwhile, funders of transparency and open data initiatives could incentivize producers to publicly disclose development datasets that are in highest demand and advise them on implementing open data principles in practice.

Producers have real concerns regarding the costs of data collection and the risks of increased public scrutiny. Funders could neutralize these deterrents to openness through helping producers develop sustainable business models and data management practices that cover costs and reduce their liability.

Section 6.1.2.2
Principle #5: Producers should partner with infomediaries – media, civil society, and think tanks – to promote data and translate it into actionable insights

Producers sometimes succumb to a narrow view of their mandate as limited to the collection, processing, and publication of raw data. To ensure their data products are being used, they must embrace a larger mandate for themselves as seeking “to translate raw [data] into actionable insights” (Custer et al., 2016, p.60). Producers also need an intentional dissemination strategy above and beyond publishing a dataset or holding a one-off launch event for a data portal. Otherwise, they are vulnerable to a ‘build it and they will come’ fallacy.

But data producers may not be well positioned to directly reach prospective target users outside of their organizations and agencies. Infomediaries (or information intermediaries) such as the media, community-based organizations, think tanks, and professional associations could play a complementary role in translating and communicating development data via various online and offline channels (Read and Atinc, 2017).

In brokering partnerships with such infomediaries, producers could adopt a hybrid dissemination strategy, as recommended by Gigler et al. (2014), to share development data that blends “high-tech” (e.g., Internet, social media, mobile phones) and “low-tech” (e.g., radio, print, community meetings) communications. This multi-channel approach is particularly important for data producers to overcome Internet connectivity constraints in low-bandwidth environments or in contexts where there is a low rate of Internet penetration.

Custer et al. (2016) spotlight an example of what these proactive producer-infomediary partnerships could look like in practice. Under the leadership of former Secretary Cesar Purisima, the Philippines Department of Finance (the data producer) collaborated with several media outlets (the infomediaries) to spark public interest in curbing tax evasion. The government’s signature Tax Watch campaign uniquely went beyond publishing raw data to placing “weekly full-page ads in domestic newspapers [in print and online] that combine killer statistics, intuitive infographics, and compelling visuals” (p.60).

Producers also need better intelligence on which channels are most effective to reach different user groups. Masaki et al. (2016), in their study of governance data use, find that a one-size fits all strategy may not be possible as “the channels that leaders use to find governance data vary by where leaders work” (p.10). AidData and its consortium partners in the AidData Center for Development Policy are shedding further light on the efficacy of different outreach modalities by testing the responses of prospective users of the aid information management system in Timor-Leste to a 3-minute video spot introducing the system’s functionalities and an email newsletter with analysis and visuals from the system.155

Section 6.1.2.3
Principle #6: Training will be critical to overcoming the capacity gap, but funders and producers must be more systematic in testing what does and does not work

Even when government, development partner, and civil society organization representatives are aware of the data available to them, they may still lack the capacity to confidently access and use this information. As Paris21 (2016) argues, equipping staff in national statistics offices and other domestic institutions with the skills to “identify, evaluate, and access new data sources” will be critical to realizing the data revolution for sustainable development. They also offer some promising ideas for how to serve multiple audiences simultaneously, such as: setting up centers of excellence to pool training resources, publishing Big Data sandboxes for people to explore relevant data sources and tutorials, and leveraging partnerships with academia and the private sector to support curricula design and deliver training (Paris21, 2016).

Training is also an underutilized channel for individual producers to increase awareness of their data products and solicit feedback from prospective users. Interviewees across the three countries frequently expressed interest in receiving additional training in the use and analysis of data to support their work. If generic launch events and media coverage allow producers to broadly advertise their data to a general audience, in theory, customized skills trainings and focused consultations are an opportunity to go deep with key user groups.

While prospective users have a clear interest in more training opportunities, not all training may be equally effective in promoting long-term data use. There are many critical decisions that go into the design of a one-off training event or a longer-term capacity-building program – who participates, how is training delivered, what mechanisms are in place to ensure follow-up, to name a few. Surprisingly little is known about the attributes of training and capacity-building efforts that are most likely to succeed.156

AidData and its consortium partners in the AidData Center for Development Policy are helping to close this evidence gap through conducting field experiments in Timor-Leste and Honduras to assess whether and how participation in a training event changes usage patterns of the aid information management system among domestic policymakers and practitioners.157 Funders, data producers, and researchers should build upon this evidence base and conduct additional experiments to test how users respond to different formulations of data training and statistical capacity programs to improve the returns on their investments.

Section 6.1.3
Choice

The mere existence of development data is not transformative. Policymakers and practitioners must choose to act upon this information. To counteract the idea that using data is optional, government, civil society, and development partner leaders must create the conditions that reward those who make decisions based upon evidence and penalize those who do not.

Section 6.1.3.1
Principle #7: Producers of data solutions should seek to enhance, not replace informal knowledge sharing via social networks

Government, development partner, and civil society organization (CSO) leaders have a natural tendency to source information from those they know due to the confluence of habit, preference, and lack of perceived alternatives. Producers of data solutions should attempt to support these robust human information networks, not replace or compete with them. Specifically, they could proactively enlist the support of a sector working group or an association of CSOs to adopt centralized (and open) data systems in a mutually beneficial alliance.

Producers could collaborate with network members to ensure that their databases are capturing up-to-date development cooperation activities for all actors. As network members become more involved in curating and adding to central data repositories, this is more likely to increase the confidence of prospective users in the comprehensiveness and utility of government data. In addition, producers could seek out the assistance of sector working groups or well-respected professional associations as third-party validators to publicly certify or give their seal of approval for the data’s accuracy.

Sector working groups or professional associations could partner with official data producers for help in documenting qualitative lessons learned for future use. Official data producers could systematically integrate this information as a value addition in centralized repositories that informal working groups or associations do not have to maintain. As official producers grow in their data management capacity, they could provide critical assistance to networks with new tools, methods, and approaches to organizing qualitative insights that are often unstructured, subjective, and context-specific.

Section 6.1.2.3
Principle #8: Funders and producers should work with the grain of human behavior to align incentives in favor of data-driven decision-making

For many government, development partner, and CSO leaders, analysis of data and its use in the context of decision-making are viewed as “overtime activities” (Development Gateway, 2016, p.3). This response is unfortunate, but understandable, as policymakers and practitioners are guided by an internal cost-benefit calculus. The informal norms and formal rules governing decision-making in government agencies or organizations seldom reward those who make data-driven decisions or penalize those that do not. Therefore, the perceived costs in time, effort, or lost political capital to using data are known and proximate, while the perceived benefits of making decisions based upon evidence are uncertain and distant.

Regardless of whether one is a skeptic of, or advocate for, the post-2015 development agenda, it is clear that this process offers a compelling window of opportunity to change the “rules of the game” (North, 1990, p.3). If funders and producers of data and statistics want to create the conditions for evidence-informed policymaking, they need to find a way to “work with the grain” of human behavior (Levy, 2014, p.7).158 In other words, they must crowd in, rather than short-circuit, the interest of political actors in favor of using data as they allocate resources, target projects, and evaluate the performance of development programs (Custer et al., 2016).

Monetary and non-monetary rewards should be employed to increase the perceived benefits for government, development partner, and civil society leaders to use development data. Executive branch-led transparency initiatives in countries such as the Philippines have tied performance bonuses and merit increases to adherence with specified disclosure practices (Custer et al., 2016). Funders and producers of development data could work with senior leaders in forward-leaning government agencies and organizations to develop a corollary to tie performance bonuses to behaviors that exhibit effective use of data to inform decision-making. In addition, senior leaders could explicitly reinforce cases of positive deviance within their agencies and organizations by prioritizing access to training, recognition, and advancement opportunities for decision-makers who use development data in their work (Zeitlin et al., 1990; Development Gateway, 2016).159

Section 6.1.3.3
Principle #9: Producers should be more transparent about their data collection methods and quality assurance procedures to get a credibility boost

The determinations that policymakers and practitioners make, regarding whether a given data source is credible, are critical to whether they ultimately decide to use this information in their work (Masaki et al., 2016). Unfortunately, prospective users perceive many data sources, particularly those produced by governments in low- and middle-income countries, to be untrustworthy and unreliable. As previously discussed, ensuring the accuracy of development data is indeed a problem. However, there is likely more behind this trust deficit than a lack of accuracy alone. Distrust of official data sources could also reflect low public confidence in the government overall.

Opaque methods for collection and validation do little to alleviate concerns that producers are manipulating data behind the scenes for their own political ends. Increasing public trust in development data will require not only technical fixes to improve the accuracy of the data, but also a political commitment to greater openness and transparency about what is being measured, how data is collected, and what procedures protect this information from tampering. If credibility is indeed a major barrier to use, data producers, both official and unofficial, have everything to gain from becoming more transparent in documenting their assumptions, methods, and processes for managing data.

Funders of data initiatives and senior leaders in governments and organizations should reward and reinforce the efforts of producers to become more transparent in their methods and assumptions. In doing so, they will help create the conditions for more robust data use. Public perceptions may lag behind policy changes, but transparent producers are more likely than their stubbornly opaque counterparts to win back the confidence of prospective data users.

Section 6.2
Final Thoughts

With the signing of Agenda 2030 in September 2015, the international community shifted from deliberating the content of the Sustainable Development Goals (SDGs) to determining how to operationalize them in practice at global, national, and subnational levels. This has spurred a broader discussion regarding the capacity that countries will need to collect, process, and disseminate “an unprecedented amount of data and statistics…[from] multiple stakeholders” (HLG-PCCB, 2017, p. 1).

The “data revolution” has put statistics squarely at the heart of the sustainable development agenda. Success will depend upon not only the international community’s ability to transform how we produce data, but also the extent to which governments, civil society, and development partners use this information effectively to improve people’s lives. Technical efforts to strengthen data production are laudable, but they will have little utility if divorced from the messiness of the decision-making processes, in which policymakers and practitioners make consequential choices about how to translate the rhetoric of the SDGs into reality.

This report, while not exhaustive, offers actionable intelligence on what policymakers and practitioners want from development data. We hope that the country-specific findings and broader operating principles outlined in this study will help funders and producers of the next generation of data and statistics to avoid data graveyards and create the conditions that make decisions based upon evidence the norm, rather than the exception.

References

References for Chapter 1

Badiee, S., Swanson, E., Eele, G., & Espey, J. (2016). State of Development Data Funding. Retrieved from: http://opendatawatch.com/wp-content/uploads/2016/09/development-data-funding-2016.pdf

Custer, S., Rice, Z., Masaki, T., Latourell, R., & Parks, B. (2015). Listening to Leaders: Which Development Partners Do They Prefer and Why? Williamsburg, VA: AidData. Retrieved from: http://aiddata.org/listening-to-leaders.

Custer, S. (2014, October 29). Avoiding Data Graveyards: 5 Questions the Data Revolution Should Be Asking. [Web log post] The First Tranche. Retrieved from: http://aiddata.org/blog/avoiding-data-graveyards-5-questions-the-data-revolution-should-be-asking

Custer, S., Xu, Q., Brigham, C., & Rahemtulla, H. (2012). The Open Data ‘Bazaar’: Crowdsourcing Solutions to Improve Data Accuracy and Re-Use in Kenya, unpublished draft.

Custer, S., Rahemtulla, H., Kaiser, K., and van den Brink, R. (2016). From Pork to Performance: Open Government and Program Performance Tracking in the Philippines. World Bank and AidData. Retrieved from: http://documents.worldbank.org/curated/en/174491467602317699/Pork-to-performance-open-government-and-program-performance-tracking-in-the-Philippines-phase-two

Davies, T. (2014). Open Data in Developing Countries - Emerging Insights from Phase I. Web Foundation. Retrieved from: http://www.opendataresearch.org/sites/default/files/publications/Phase%201%20- %20Synthesis%20-%20Full%20Report-print.pdf

Development Gateway (2016). Increasing the Impact of Results Data. Policy Brief. Development Gateway. Retrieved from: http://www.developmentgateway.org/assets/post-resources/RDI-PolicyBrief.pdf

The Economist (2015). The economics of optimism: The debate heats up about what goals the world should set itself for 2030. The Economist. Retrieved from: http://www.economist.com/news/finance-and-economics/21640361-debateheats-up-about-what-goals-world-should-set-itself-2030

Global Partnership for Sustainable Development Data (GPSDD) (2016). About Us: Who We Are. Global Partnership for Sustainable Development Data. Retrieved from: http://www.data4sdgs.org/who-we-are/.

Jerven, M. (2014). Benefits and Costs of the Data for Development Targets for the Post-2015 Development Agenda. Post-2015 Consensus Working Paper. Retrieved from: http://www.copenhagenconsensus.com/sites/default/files/data_assessment_-_jerven.pdf

Jerven, M. (2016, July 22). Governing the world ‘as if’ it counts. DevEx. Retrieved from: https://www.devex.com/news/governing-the-world-as-if-it-counts-88437

Khan, S., & Foti, J. (2015). Aligning Supply and Demand for Better Governance: Open Data in the Open Government Partnership. Open Government Partnership. Retrieved from: http://www.opengovpartnership.org/sites/default/files/IRMReport-OpenData_0.pdf

Parks, B., Rice, Z., and Custer, S. (2015). The Marketplace of Ideas for Policy Change: Who do developing world leaders listen to and why? Williamsburg, VA: AidData and the College of William and Mary. Retrieved from: http://aiddata.org/marketplace-of-ideas-for-policy-change

Post-2015 Data Test (2014). Presenting Initial Findings from the Post-2015 Data Test. [Web log post] Post-2015 Data Test. Retrieved from: http://www.post2015datatest.com/event/presenting-initial-findings-post-2015-data-test/

Read, L., & Atinc, T. (2016, December 21). From data to learning: the role of social accountability in education systems. [Web log post] Brookings Institution. Retrieved from: https://www.brookings.edu/blog/education-plus-development/2016/12/21/from-data-to-learning-the-role-of-social-accountability-in-education-systems/

United Nations (2014). A World That Counts: Mobilising the Data Revolution for Sustainable Development. United Nations. Retrieved from: http://www.undatarevolution.org/wp-content/uploads/2014/11/A-World-That-Counts.pdf

United States Agency for International Development (2015). Aid Transparency Country Pilot Assessment: Final Report. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1870/AidTransparencyCountryPilotAssessment.pdf

Urbina, I., & Nixon, R. (2007, March 30). Disuse of System Is Cited in Gaps in Soldiers’ Care. New York Times. Retrieved from: http://www.nytimes.com/2007/03/30/washington/30medical.html

References for Chapter 2

Bowen, S., & Zwi, A. B. (2005). Pathways to “Evidence-Informed” Policy and Practice: A Framework for Action. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1140676/

Canares, M., & Shekhar, S. (2015). Open Data and Sub-national Governments: Lessons from Developing Countries. Retrieved from: http://webfoundation.org/docs/2015/08/ODDC-Phase-2-Paper- Subnational.pdf

Custer, S., Rice, Z., Masaki, T., Latourell, R., & Parks, B. (2015). Listening to Leaders: Which Development Partners Do They Prefer and Why? Williamsburg, VA: AidData. Retrieved from: http://aiddata.org/listening-to-leaders

Custer, S., Rahemtulla, H., Kaiser, K., & van den Brink, R. (2016a). From Pork to Performance: Open Government and Program Performance Tracking in the Philippines. World Bank and AidData. July 2016. Retrieved from: http://documents.worldbank.org/curated/en/174491467602317699/Pork-to-performance-open-government-and-program-performance-tracking-in-the-Philippines-phase-two

Custer, S., Masaki, T., Sethi, T., Latourell, R., Rice, Z., & Parks, B. (2016b). Governance Data: Who Uses It and Why? Williamsburg, VA: AidData and the Governance Data Alliance. Retrieved from: http://aiddata.org/governance-data-report

Davies, T. (2014). Open Data in Developing Countries - Emerging Insights from Phase I. Web Foundation. Retrieved from: http://www.opendataresearch.org/sites/default/files/publications/Phase%201%20-%20Synthesis%20-%20Full%20Report-print.pdf

Devarajan, S. (2011). Africa’s statistical tragedy. [Web log post] World Bank. Retrieved from: http://blogs.worldbank.org/africacan/africa-s-statistical-tragedy

Development Gateway (2016). Increasing the Impact of Results Data. Policy Brief. Development Gateway. Retrieved from: http://www.developmentgateway.org/assets/post-resources/RDI-PolicyBrief.pdf

Dhaliwal, I. (2015). Barriers to Evidence-Informed Decision-making: Towards Globally Informed, Locally Grounded Policymaking. Retrieved from: https://www.povertyactionlab.org/sites/default/files/L7_Generalizability-Iqbal-v7.pdf

Fox, J. (2014). Social Accountability: What Does the Evidence Really Say? Global Partnership for Social Accountability. Working Paper No. 1. World Bank.

Gigler, S., Custer, S., & Rahmetulla, H. (2011). Realizing the Vision of Open Government Data- Opportunities, Challenges and Pitfalls. Open Development Technology Alliance, The World Bank.

Gigler, B., Custer, S., Bailur, S., Dodds, E., Asad, S., & Gagieva-Petrova, E. (2014). Closing the Feedback Loop: Can Technology Amplify Citizen Voices? In B.-S. Gigler, and S. Bailur (Eds.), Closing the Feedback Loop: Can Technology Bridge the Accountability Gap? World Bank.

Haythornthwaite, C. (1996). Social network analysis: An approach and technique for the study of information exchange. Retrieved from: http://www.sciencedirect.com/science/article/pii/S0740818896900031 doi: 10.1016/S0740-8188(96)90003-1

IDB (2011). IDB will support new national census of population and housing in Honduras. News release. Retrieved from: http://www.iadb.org/en/news/news-releases/2011-06-01/national-census-of-population-and-housing-in-honduras,9396.html

Kerner, A., Jerven, M., & Beatty, A. (2015). Does it Pay to be Poor? Testing for Systematically Underreported GNI estimates. Review of International Organizations, 1­38.

Khemani, S., Bo, E.D., Ferraz, C., Finan, F., Stephenson, C., Odugbemi, A., Thapa, D., & Abrahams, S. (2016). Making Politics Work for Development: Harnessing Transparency and Citizen Engagement. Policy Research Report, World Bank, Washington, DC. Retrieved from: http://documents.worldbank.org/curated/en/268021467831470443/Making-politics-work-for-development-harnessing-transparency-and-citizen-engagement

Kosack, S., & Fung, A. (2014). Does Transparency Improve Governance? Annual Review of Political Science, 17, 65-87.

Masaki, T., Sethi, T., & Custer, S. (2016). In the Eye of the Beholder: When is governance data “good enough”? Williamsburg, VA. AidData at the College of William & Mary and the Governance Data Alliance.

OECD (2012). The Busan Partnership for Effective Development Cooperation. July 2012. Retrieved from: https://www.oecd.org/dac/effectiveness/Busan%20partnership.pdf

PARIS21 (2016). Partner Report on Support to Statistics. September 2016. Retrieved from: http://www.paris21.org/sites/default/files/PRESS-2016-web-final.pdf

Parks, B., Rice, Z., & Custer, S. (2015). The Marketplace of Ideas for Policy Change: Who do developing world leaders listen to and why? Williamsburg, VA: AidData and The College of William and Mary. Retrieved from: http://aiddata.org/marketplace-of-ideas-for-policy-change

Peixoto, T. (2013). The Uncertain Relationship between Open Data and Accountability: A Response to Yu and Robinson’s The New Ambiguity of Open Government. UCLA Law Review, 200 (60), 202-213.

Pew-MacArthur (2014). Evidence-Based Policymaking: A guide for effective government. A report from the Pew-MacArthur Results First Initiative. November 2014. Pew Charitable Trusts and MacArthur Foundation. Retrieved from: http://www.pewtrusts.org/~/media/assets/2014/11/evidencebasedpolicymakingaguideforeffectivegovernment.pdf

Rahemtulla, H., Kaplan, J., Gigler, S., Custer, S., Keiss, J., & Brigham, C. (2011). Open Data Kenya. In: World Bank, Open Development Technology Alliance Working Paper Series No. 5. Retrieved from https://www.scribd.com/document/75642393/Open-Data-Kenya-Long-Version

Read, L., & Atinc, T. (2017). Information for accountability: Transparency and citizen engagement for improved service delivery in education systems. Brookings Institution. Retrieved from: https://www.brookings.edu/wp-content/uploads/2017/01/global_20170125_information_for_accountability.pdf

Sandefur, J., & Glassman, A. (2014). The Political Economy of Bad Data: Evidence from African Survey and Administrative Statistics. Center for Global Development Working Paper 373. Center for Global Development. Retrieved from: https://www.cgdev.org/sites/default/files/political-economy-bad-data.pdf

Stewart, R. (2014). A theory of change for capacity building for the use of research evidence by decision makers in southern Africa. Policy Press. Retrieved from: http://www.ingentaconnect.com/content/tpp/ep/pre-prints/content-EvP_037

United Nations (2014). A World That Counts: Mobilising the Data Revolution for Sustainable Development. United Nations. Retrieved from: http://www.undatarevolution.org/wp-content/uploads/2014/11/A-World-That-Counts.pdf

United Nations (2015). MDG Report 2015. Retrieved from: http://www.un.org/millenniumgoals/2015_MDG_Report/pdf/MDG%202015%20rev%20(July%201).pdf

UNDP (2011). Africa Forum on Civil Society and Governance Assessments. 10-12 November 2011, Dakar, Senegal. Retrieved from: http://www.undp.org/content/dam/undp/documents/partners/civil_society/additional_documents/Africa%20Forum%20on%20Civil%20Society%20and%20Governance%20Assessments/Africa%20Regional%20Civil%20Society%20and%20Governance%20Workshop%20-%2028%20September.pdf

UNSTATS (2001). Symposium on Global Review of 2000 Round of Population and Housing Censuses: Mid-Decade Assessment and Future Prospects. Statistics Division, Department of Economic and Social Affairs, United Nations Secretariat. New York, 7-10 August. Retrieved from: http://unstats.un.org/unsd/demog/docs/symposium_03.htm#_Toc7427914

United States Agency for International Development (2015). Aid Transparency Country Pilot Assessment: Final Report. Retrieved October 05, 2016, from: https://www.usaid.gov/sites/default/files/documents/1870/AidTransparencyCountryPilotAssessment.pdf

Wilson, C., & Rahman, Z. (n.d.). Citizen-Generated Data and Governments: Towards a Collaboration. Retrieved from: http://civicus.org/images/citizen-generated%20data%20and%20governments.pdf

World Bank (2012, December 20). Timor-Leste’s telecommunications sector will soon help more people access to a variety of services. Retrieved from: http://www.worldbank.org/en/news/feature/2012/12/20/competition-in-timor-leste-telecommunications-sector-will-help-more-people-access-to-a-variety-of-services

World Bank (2016). World Development Report 2016: Digital Dividends. Washington, DC: World Bank.

References for Chapter 3

Gigler, S., Custer, S., & Rahmetulla, H. (2011). Realizing the Vision of Open Government Data – Opportunities, Challenges and Pitfalls, Open Development Technology Alliance, The World Bank. Retrieved from: https://www.scribd.com/collections/3389411/Open-Development-Technology-Alliance

IDB (2000). Central America After Hurricane Mitch: The Challenge of Turning a Disaster into an Opportunity, Inter-American Development Bank.

IDB (2011). IDB will support new national census of population and housing in Honduras. News release. Retrieved June 1, 2016 from: http://www.iadb.org/en/news/news-releases/2011-06-01/national-census-of-population-and-housing-in-honduras,9396.html

INE (2014). Línea de Pobreza 2015. Retrieved from: http://www.ine.gob.hn/index.php/component/content/article?id=91

LaPrensa (2014, January 3). Juan Orlando Hernández gobernará con un “superministro” y 7 sectoriales. Retrieved from: http://www.laprensa.hn/honduras/tegucigalpa/440979-98/juan-orlando-hernandez-gobernara-con-un-superministro-y-7-sectoriales

Ministry of Foreign Affairs (2010). Summary of Contents of Decree Law N. 286-2009 on the Establishment of a Country Vision and Adoption of a Nation Plan to Honduras. Retrieved from: http://www.hondurasemb.org/boletines-de-prensa/9_septiembre_2010/Summary_Country_Vision_and_Adoption_Nation_Plan.pdf

OECD (2009). The human development index as an effort to measure well-being in Honduras. The 3rd OECD Forum on Statistics, Knowledge and Policy. Busan, Korea, 27-30 October. Retrieved from: http://www.oecd.org/site/progresskorea/44110008.pdf

OECD (2014). Aid statistics by donor, recipient and sector. Retrieved from: http://www.oecd.org/statistics/datalab/oda-recipient-sector.htm

OECD (2016). Net ODA (indicator). doi: 10.1787/33346549-en. Retrieved May 31, 2016.

Pino, H. N. (2012). The Impact of the International Budget Partnership’s Open Budget Survey and its Partner Institutions’ Advocacy on Budget Transparency in Honduras. ICEFI. Retrieved from: http://www.internationalbudget.org/wp-content/uploads/OBI-case-study-Honduras.pdf

SCGG (2014). Dirección Presidencial de Planificación Estratégica, Presupuesto e Inversión Pública. Retrieved from: http://www.scgg.gob.hn/content/planificacion-estrategica-presupuesto-e-inversion-publica

Transparency International (2014, October 6). Honduras Government, ASJ, and Transparency International sign agreement for transparency. Press release. Retrieved from: http://www.transparency.org/news/pressrelease/honduras_government_asj_and_transparency_international_sign_agreement_for_t

United States Department of State (2009). Termination of Assistance and Other Measures Affecting the De Facto Regime in Honduras. Press statement. Retrieved from: http://www.state.gov/r/pa/prs/ps/2009/sept/128608.htm

UNODC (2013). Global Study on Homicide 2013. Retrieved from: https://www.unodc.org/documents/gsh/pdfs/2014_GLOBAL_HOMICIDE_BOOK_web.pdf

References for Chapter 4

Katy, A. (2007). Assessment of the Effectiveness of Literacy and Numeracy Programs in Timor-Leste. Retrieved from: http://pdf.usaid.gov/pdf_docs/pdacx743.pdf

Pereira, Á. (2012, April 24). Government Results Portal Launched [Press Release]. Retrieved from: http://www.laohamutuk.org/Oil/EITI/2012/SSCOMResultsPortal24Apr12En.pdf

Australian Department of Foreign Affairs and Trade (2016). Timor-Leste Aid Program: Monitoring and Evaluation House, Buka Hatene. Retrieved from: http://dfat.gov.au/about-us/business opportunities/tenders/Documents/timor-leste-m-e-house-final-design.pdf

East Timor and Indonesia Action Network (2008, July 14). Still No Justice in East-Timor War Crimes. Retrieved from: https://www.globalpolicy.org/component/content/article/165/29635.html

Fragility Assessment Team at Ministry of Finance (2013, February 26). Summary Report: Fragility Assessment in Timor-Leste. Retrieved from: http://www.g7plus.org/sites/default/files/resources/Timor-Leste-Fragility-Assessment-Report.pdf

Freedom House (2015). Freedom of the Press: East Timor. Retrieved from: https://freedomhouse.org/report/freedom-press/2015/east-timor

g7+ (2016). New Deal Implementation. Retrieved from: http://www.g7plus.org/en/our-work/new-deal-implementation

Government of Timor-Leste (2011). Timor-Leste Strategic Development Plan 2011-2030. Retrieved from: https://sustainabledevelopment.un.org/content/documents/1506Timor-Leste-Strategic-Plan-2011-20301.pdf

Government of Timor-Leste (2013). State Budget 2014 Budget Overview Book 1. Retrieved from: https://www.mof.gov.tl/wp-content/uploads/2013/10/FINAL_20131024_Budget_Book_1_EN_16.10.pdf

Government of Timor-Leste Ministry of Finance Development Partnership Management Unit (2014). 2014 Development Cooperation Report for Timor-Leste: Overview. Retrieved from: https://www.mof.gov.tl/wpcontent/uploads/2011/07/merged_document.pdf

Government of Timor-Leste (2015). Council of Ministers of the VI Constitutional Government. Retrieved from: http://timor-leste.gov.tl/?cat=38&lang=en

Government of Timor-Leste Ministry of Finance (2016). Millennium Development Goals - 2015 Timor-Leste. Retrieved from: https://www.mof.gov.tl/wp-content/uploads/2016/04/Millennium_Development_Goals_2015_Timor_Leste.pdf

La’o Hamutuk (2015, May 25). 2016 General State Budget. Retrieved from: http://www.laohamutuk.org/econ/OGE16/15OGE16.htm

Lundhal, M. & Sjöholm, F. (2013). Improving the Lot of the Farmer: Development Challenges in Timor-Leste during the Second Decade of Independence. Asian Economic Papers 12:2. Retrieved from: http://www.mitpressjournals.org/doi/pdf/10.1162/ASEP_a_00211

Marx, S. & Pinhero, M.F.C. (2013, September 4). The Silent Crisis in Timor-Leste’s Development Trajectory. Retrieved from: http://asiafoundation.org/in-asia/2013/09/04/the-silent-crisis-in-timor-lestes-development-trajectory/

Ministry of Agriculture and Fisheries’ Agricultural Rehabilitation Project III (ARP III) (2008). Policy Framework for Agricultural Extension in Timor-Leste. December. Retrieved from: http://www.g-fras.org/en/2015-05-28-15-50-27/australia-20.html?download=329:policy-framework-for-agricultural-extension-in-timor-leste

National Statistics Directorate Ministry of Finance (2010). The Statistical Work Plan for Timor-Leste 2010-2019. June. Retrieved from: http://www.paris21.org/sites/default/files/TIMORLESTE_SWP2010-2019.pdf

Rajalingam, G. (2014). 2014 Survey of Travelers to Timor-Leste. Retrieved from: http://asiafoundation.org/resources/pdfs/VisitorSurvey2014English.pdf

Revenue Watch Institute and Transparency International (2010). 2010 Revenue Watch Index. Retrieved from http://www.revenuewatch.org/rwindex2010/pdf/RevenueWatchIndex_2010.pdf

Ruak, T.M. (2015, December 28). Message to the National Parliament from the President of the Democratic Republic of Timor-Leste, Taur Matan Ruak, with the Grounds for the Veto of the 2016 General State Budget (National Parliament Decree no. 20/III, 2016 General State Budget). Retrieved from: http://www.laohamutuk.org/econ/OGE16/veto/PresLetter29Dec2015en.pdf

UNDP (2012). About Timor-Leste. Retrieved from: http://www.tl.undp.org/content/timor_leste/en/home/countryinfo.html

UNICEF (2013, December 31). At a glance: Timor-Leste. Retrieved from: http://www.unicef.org/infobycountry/Timorleste_statistics.html

World Bank (2012, December 20). Timor-Leste’s telecommunications sector will soon help more people access to a variety of services. Retrieved from: http://www.worldbank.org/en/news/feature/2012/12/20/competition-in-timor-leste-telecommunications-sector-will-help-more-people-access-to-a-variety-of-services

World Bank (2014). Internet Users (per 100 people) [Data file]. Retrieved from: http://data.worldbank.org/indicator/IT.NET.USER.P2?locations=TL

References for Chapter 4

All Africa (2015). Emerging Senegal Plan. Retrieved from: http://allafrica.com/infocenter/PSE_2015/

Ba, D. (2016, March 14). Retrieved from: http://www.theafricareport.com/West-Africa/senegals-economic-plan-drives-growth.html

Issoufou, S., & Kireyev, A. (2015, June 25). New Plan Sets Senegal On Course to Become Emerging Economy. Retrieved from: http://www.imf.org/external/pubs/ft/survey/so/2015/car062415a.htm

Kireyev, A. (2014, December 24). New Plan Sets Senegal On Course to Become Emerging Economy. Retrieved from: http://www.imf.org/external/pubs/ft/survey/so/2014/CAR122414A.htm

Mauru, R. (2010). Demographic Data for Development - Senegal. Retrieved from: http://www.hewlett.org/uploads/files/Senegalfinalreport.pdf

Petesch, C. (2016, March 20). Senegal votes on referendum for sweeping constitutional reforms including reduction of presidential term from 7 to 5 years. Retrieved from: http://www.usnews.com/news/world/articles/2016-03-20/senegal-votes-on-referendum-to-reduce-presidential-term

Pham, J. P. (2016, March 29). Senegal Consolidates Its Constitutional Democracy. Retrieved March 20, 2017, from http://www.atlanticcouncil.org/blogs/africasource/senegal-consolidates-its-constitutional-democracy

Poverty Reduction Strategy Papers (PRSP) (2013, July 3). Retrieved from: http://www.imf.org/external/np/prsp/prsp.aspx#HeadingS

Sakho-Jimbira, M. S., Hathie, I., Wade, I., Niang, A., & Niang, M. (2015, December 1). Measuring Sustainable Development for Post-2015 in Senegal. Retrieved from: http://www.post2015datatest.com/wp-content/uploads/2015/12/Final-Version-Senegal-Report-Data-Test.pdf

USAID. (2016, November 20). Strengthening Democratic Governance in Senegal | Fact Sheet | Senegal. Retrieved March 20, 2017, from https://www.usaid.gov/senegal/fact-sheets/strengthening-democratic-governance-senegal

World Bank (2016, April 21). Country Overview - Senegal. Retrieved from: http://www.worldbank.org/en/country/senegal/overview

References for Chapter 6

Badiee, S., Swanson, E., Eele, G., & Espey, J. (2016). State of Development Data Funding. Retrieved from: http://opendatawatch.com/wp-content/uploads/2016/09/development-data-funding-2016.pdf

Custer, S., Rahemtulla, H., Kaiser, K., & van den Brink, R. (2016). From Pork to Performance: Open Government and Program Performance Tracking in the Philippines. World Bank and AidData. July 2016. Retrieved from: http://documents.worldbank.org/curated/en/174491467602317699/Pork-to-performance-open-government-and-program-performance-tracking-in-the-Philippines-phase-two

Custer, S., Xu, Q., Brigham, C., & Rahemtulla, H. (2012). The Open Data ‘Bazaar’: Crowdsourcing Solutions to Improve Data Accuracy and Re-Use in Kenya, unpublished draft.

Devarajan, S. (2011). Africa’s statistical tragedy. [Web log post] World Bank. Retrieved from: http://blogs.worldbank.org/africacan/africa-s-statistical-tragedy

Development Gateway (2016). Increasing the Impact of Results Data. Policy Brief. Development Gateway. Retrieved from: http://www.developmentgateway.org/assets/post-resources/RDI-PolicyBrief.pdf

Gigler, B.S., Custer, S., Bailur, S., Dodds, E., Asad, S., & Gagieva-Petrova, E. (2014). Closing the Feedback Loop: Can Technology Amplify Citizen Voices? In B.-S. Gigler, and S. Bailur (Eds.), Closing the Feedback Loop: Can Technology Bridge the Accountability Gap? World Bank.

HLG-PCCB. (2017). Cape Town Global Action Plan for Sustainable Development Data. High-level Group for Partnership, Coordination, and Capacity-Building (HLG-PCCB) for statistics for the 2030 Agenda for Sustainable Development. Cape Town, South Africa. 15 January 2017. Retrieved from: http://undataforum.org/WorldDataForum/wp-content/uploads/2017/01/Cape-Town-Action-Plan-For-Data-Jan2017.pdf

Levy, B. (2014). Working with the Grain: Integrating Governance and Growth in Development Strategies. Oxford University Press: New York, NY. Kindle edition.

Masaki, T., Sethi, T., & Custer, S. (2016). In the Eye of the Beholder: When is governance data “good enough”? Williamsburg, VA. AidData at the College of William & Mary and the Governance Data Alliance.

North, D. (1990). Institutions, Institutional Change, and Economic Performance (Political Economy of Institutions and Decisions. Cambridge University Press. Kindle edition.

Paris21 (2016). NSDS Guidelines. Data Revolution. Retrieved from: http://nsdsguidelines.paris21.org/node/716

Paris21 (n.d.). How to Build Statistical Capacity. Capacity Building Knowledge Series. Retrieved from: https://www.paris21.org/sites/default/files/Capacity-buiding-knowledge-series.pdf

Read, L. and T. Atinc. (2017). Information for Accountability: Transparency and Citizen Engagement for Improved Service Delivery in Education Systems. Brookings Institution. Retrieved from: https://www.brookings.edu/wp-content/uploads/2017/01/global_20170125_information_for_accountability.pdf

Sandefur, J., & Glassman, A. (2014). The Political Economy of Bad Data: Evidence from African Survey and Administrative Statistics. Center for Global Development Working Paper 373. Center for Global Development. Retrieved from: https://www.cgdev.org/sites/default/files/political-economy-bad-data.pdf

Stuart, E., Samman, E., Avis, W., & Berliner, T. (2015). The Data Revolution: Finding the Missing Millions. ODI Research Report #3. Retrieved from https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9604.pdf

Zeitlin, M., Ghassemi, H., & Masour, M. (1990). Positive Deviance in Child Nutrition - with emphasis on Psychosocial and Behavioural Aspects and Implications for Development. The United Nations University: Tokyo, Japan. Retrieved from:http://www.positivedeviance.org/pdf/publication%20nutrition/1990%20zeitlin%20posdev.pdf


1 This introduction builds upon an October 2014 article by Samantha Custer on AidData.org, “Avoiding Data Graveyards: 5 Questions the Data Revolution Should Be Asking.”

2 The Sustainable Development Goals (SDGs) agenda has called for a “data revolution:” more and better data to be able to monitor progress on the 17 goals, 169 targets and 230 indicators.

3 Author’s notes from the proceedings. Ms. Mohammed is Deputy Secretary General of the United Nations and formerly served as Nigeria’s Minister of Environment, among other positions.

4 Also see Morten Jerven’s post “Governing the world ‘as if’ it counts” from July 22, 2016 in DevEx https://www.devex.com/news/governing-the-world-as-if-it-counts-88437.

5 This ToC is explained and unpacked in greater depth in Chapter 2.

6 The estimate projected by Badiee et al. (2016) is for low- and middle-income countries and is based on the costs to produce data to monitor a subset of SDG indicators categorized as Tier 1 (methods are established and data is collected somewhat regularly) or Tier 2 (methods are established but data is not regularly collected). The 80 indicators categorized as Tier 3 (no methods established) are not included, implying that the actual shortfall in financing for data could be much larger.

7 For more information on the Global Partnership for Sustainable Development Data, see: http://www.data4sdgs.org/who-we-are/.

8 The AidData Center for Development Policy is a consortium of the College of William & Mary, Development Gateway, University of Texas-Austin, Brigham Young University, and Esri. The work of this consortium is funded by the USAID Global Development Lab through the Higher Education Solutions Network program. For more information, see: http://aiddata.org/aiddata-center-for-development-policy.

9 This literature includes well-known impact evaluations of specific information interventions such as those conducted by Bjorkman and Svensson (2009), Banerjee et al. (2010), Lieberman et al. (2013), and Bjorkman et al. (2014), as well as meta-analyses with broader scope such as those produced by Fox (2007), Khemani (2007), Fox (2014), Kosack and Fung (2014), Brockmyer and Fox (2015), and Khemani et al. (2015), to name a few. Data use studies that focus on specific sectors or categories of data are also becoming more prominent, such as those conducted by Custer et al. (2016) and Masaki et al. (2017) on governance data, Development Gateway (2016) on results data, and Read and Atinc (2017) on education data.

10 Since 2012, in collaboration with USAID’s Global Development Lab, the ACDP has partnered with the governments of 9 countries to geocode official development assistance (ODA) flows reported into their aid information management systems (AIMS). The nine countries include: the Democratic Republic of the Congo, Haiti, Honduras, Nepal, Niger, the Philippines, Senegal, Timor-Leste, and Uganda. Two more countries, Bangladesh and Ghana, are pending the start of new geocoding work.

11 The three selected countries represent three regions (sub-Saharan Africa, Latin America, Asia and the Pacific), two income levels (Senegal is low income, Honduras and Timor-Leste are lower-middle), and varying Internet connectivity (Senegal, Honduras, and Timor-Leste rank 84, 112, and 200 out of 201 countries, respectively, in terms of number of Internet users). Income-level and region information is based upon data.worldbank.org. Internet connectivity rankings are based upon internetlivestats.com. In assessing open data commitments, we looked primarily at three metrics: the existence of a national open data portal (Senegal and Timor-Leste have them), membership in the Open Government Partnership (Honduras and Senegal are members), and the presence of formal access to information legislation (only Honduras has this).

12 For the assessment of the quality of geocoded aid information, AidData examined the geocoded aid data included in a country’s aid information management system (AIMS).

13 For more information on the 2014 Reform Efforts Survey and sampling frame construction, see: aiddata.org/listening-to-leaders.

14 In the context of this analysis, we use outcomes to refer to intermediate project results data, as well as longer-term indicators of development outcomes.

15 In this study, we define a development decision as an active or tacit choice made with consequences that affect one or more of the following: allocation or prioritization of resources; approval or disapproval of project design or plans; adjustments to, or continuation of, strategies, operational plans and/or procedures; or determination of the success or failure or a project. Appendix C includes a taxonomy of decision types (grouped decisions at different stages of the policy/program/project cycle) and decision routines (formal or informal processes for decision-making in organizations/governments).

16 Appendix C also includes a taxonomy of data types and attributes that were considered in the context of the country studies.

17 This is likely true regardless of whether the data sources are produced within, or outside of, a given government or organization. Data collection is often conducted at the point of service delivery by relatively junior level staff, whereas responsibility for resource allocation and policy decisions are often centralized among senior leaders.

18 The Pew-MacArthur Results First Initiative (2014) helpfully outlines five prototypical steps in an evidence-based policymaking process: program assessment, budget development, implementation oversight, outcome monitoring, and targeted evaluation. Bowen and Zwi (2005) provide a discussion of how factors at the individual-level, organization-level, and system-level can influence a decision to accept or reject data and evidence. Presumably, these factors can undercut decision-making at all stages. See also Stewart (2014) and Dhaliwal (2015) for further discussion of common barriers to data use.

19 For the purpose of this study, we are interested in both decision-makers (individuals in a position to make consequential decisions related to development strategies, policies, programs, projects and/or activities) and decision influencers (individuals in a position to inform or influence decision-makers through provision of data and evidence, moral and political suasion, etc).

20 UNSTATS (2001) gives a helpful comparison of the three data sources used for demographic and social statistics (e.g., administrative records, surveys, and censuses). See: http://unstats.un.org/unsd/demog/docs/symposium_03.htm#_Toc7427914

21 Only Honduras has formalized access to information legislation; however, Senegal is a member of the Open Government Partnership and has signed the Open Government Declaration which includes pledges to “increase the availability of information on government activities” and “increase access to new technologies for openness and accountability.” Timor-Leste has several national open data portals on external finance (e.g., Extractives Industries Transparency Initiative, Aid Transparency Portal); expenditures (e.g., Budget Transparency Portal, eProcurement Portal); and results (e.g., Government Results Portal).

22 In the context of resourcing development projects, for example, fragmented information could cause agencies or organizations to duplicate efforts in the same communities, engage in projects that are at cross-purposes with one another, or provide ongoing funding to poor performing projects.

23 According to PARIS21’s Partner Report on Support to Statistics (PRESS) 2016, only 0.25% of ODA is dedicated to statistics. 72% of these commitments are from 5 providers: Canada, the European Commission/EUROSTAT, the FAO, the International Monetary Fund, and the World Bank. See: http://www.paris21.org/sites/default/files/PRESS-2016-web-final.pdf

24 This estimate is based upon publicly available information on the national budgets of the three countries, including: the OECD (2014) for Senegal, the 2014 Development Cooperation Report for Timor-Leste and 2015 budget for Honduras. See the corresponding country chapter for the full reference and citation.

25 There is a whole field of social science inquiry related to social network analysis: a set of techniques used to study the exchange of resources (including information) among a range of actors (individuals, groups, and organizations). See Haythornthwaite (1996): http://www.sciencedirect.com/science/article/pii/S0740818896900031

26 The Wilson and Rahman (n.d.) study defines citizen-generated data as “data that people or their organizations produce to directly monitor, demand or drive change on issues that affect them. This can be produced through crowdsourcing mechanisms or citizen reporting initiatives, often organized and managed by civil society groups. This is distinct from “big data” or social media data, which is indirectly created by citizens through interaction with social media platforms” (p.3).

27 In designing the original four C’s framework, Custer et al. (2016a) conducted an extensive review of theories of change in the literature on open data, open government, and transparency and accountability initiatives, drawing crucial insights from Khemani et al. (2016), Jonathan Fox (2014), Tiago Peixoto (2013), and Kosack and Fung (2014), among others.

28 For example, Agence Nationale de Statistique et de la Démographie (ANSD) in Senegal, Direcção Nacional de Estatística (DNE) in Timor-Leste, and Instituto Nacional de Estadística (INE) in Honduras.

29 In Timor-Leste, villages are referred to as “sucos”.

30 As Custer et al. (2016a) describe, information asymmetries and weak institutions create perverse incentives for all parties to maintain the status quo: a low accountability equilibrium.

31 Also see: Kerner et al. (2015) “Does it pay to be poor? Testing for systematically underreported GNI estimates” and Sandefur and Glassman (2014) “The Political Economy of Bad Data: Evidence from African Survey and Administrative Statistics.”

32 For example, the Sandefur and Glassman (2014) study assessed the relationship between GAVI cash incentives to low-income countries for every additional child immunized with the third dose of vaccine against diphtheria, tetanus, and measles (DTP3). In assessing discrepancies between administrative data and independent household surveys during the period before, during, and after the GAVI incentives were in effect, the authors found systematic over-reporting throughout sub-Saharan Africa.

33 For example, the Government of Honduras maintains two systems that contain information on foreign aid operating in parallel.

34 The Government of Honduras has made a series of steps to open up its budget and financial management practices, including: a commitment to the Extractives Industry Transparency Initiative, a 2014 agreement with Transparency International to increase transparency in five areas, the adoption of a Transparency and Access to Public Information Law, and creation of an Institute for Public Access to Information.

35 El Sistema de Administración de Centros Educativos or SACE.

36 For example, only two of six major National Statistics Directorate surveys are publicly available in Tetum.

37 Produced by AidData and the Governance Data Alliance, the Masaki et al. (2016) study, “When is Governance Data Good Enough?”, analyzes the responses of over 500 government, civil society, and development partner representatives in 126 low- and middle-income countries to a snap poll on their use of 29 governance data sources produced by external organizations. For more information, see: aiddata.org/governance-data-report

38 One implementing partner, unaware of the AIMS in Timor-Leste, expressed demand for a system much like it and noted that a government agency was hoping to create a database to track aid projects in the agriculture sector.

39 Some donors and international civil society organizations are not included in the AIMS reporting. As a result, the AIMS may provide an official picture of development cooperation activities that is incomplete and missing several important actors.

40 In the Masaki et al. (2016, p.19) survey of prospective governance data users, 46% of respondents said that they used governance data when it was “relevant to the questions I was trying to answer.” Meanwhile, 53% of respondents pointed to lack of relevance as a reason why they did not use a given data source.

41 The challenge of data use in the context of development cooperation practices is particularly evocative. Development partner and government representatives in all three countries express the need for better coordination and less duplication of efforts. Yet, there is little indication that these groups routinely use development cooperation data to change how they fund and site aid activities.

42 In March 2011, the GoH entered an agreement with the Millennium Challenge Corporation (MCC) to strengthen fiscal transparency and governance, using Honduras’ Public Expenditure Framework Assessment (PEFA) and the Open Budget Survey (OBS) to assess results from the new initiative. In 2012, the government used the PEFA and OBS findings in its Plan to Improve Fiscal Transparency and Management (Pino, 2012). See also: http://www.internationalbudget.org/wp-content/uploads/OBI-case-study-Honduras.pdf. Specific goals of eradicating poverty, supporting democracy, generating employment opportunities, and ensuring a transparent, responsible, and efficient State are aligned with the medium- and long-term priorities identified in two guiding documents: the Country Vision (2010-2038) and National Plan (2010-2022) (Ministry of Foreign Affairs, 2010).

43 President Juan Orlando Hernández signed a 2014 agreement with Transparency International (TI) to increase transparency and combat corruption in five areas: health, education, security, infrastructure projects, and tax administration and institution strengthening (TI, 2014 press release). The GoH signed on to several international transparency initiatives, including: the Open Government Partnership, the Open Budget Initiative, and the Extractive Industries Transparency Initiative. The GoH also adopted the Transparency and Access to Public Information Law and established the Institute for Public Access to Information.

44 Our population of interest includes development stakeholders that either make or inform decisions regarding the financing, allocation or targeting of development activities in Honduras. We held interviews with 28 government officials, 19 development partner organization staff, and 21 civil society organization staff. The findings are broadly representative of development stakeholders based in Tegucigalpa; however, they do not speak to the views of officials and staff that work at the subnational level.

45 For example, the UN funds the central education administrative system (Sistema de Administración de Centros Educativos, SACE) and UNICEF supports the National Social Sector Information Center (Centro Nacional de Información del Sector Social, CENISS). The German government also collaborated with the GoH to create a database on education infrastructure, including geo-referenced school-level data on: electricity, drinking water, latrines, and desks. This acts as a complementary data source to the secretariat of education’s SACE database that captures information on number of teachers and enrollment at the school level. Also see IDB (2011).

46 Some respondents felt that the collection of good quality data and its use was not enough of a national priority.

47 Some of this data is only available for a fee, while some is publicly available in PDF form.

48 The PGC has been implemented by the AidData Center for Development Policy partner Development Gateway.

49 The GoH uses three central documents to guide the country’s development planning: the Country Vision (2010-2038), the National Plan (2010-2022), and the four-year Government Strategic Plan that is specific to each administration.

50 The pilot initiative will integrate data from the National Social Sector Information Center (CENISS) (see Table 1). See: http://www.scgg.gob.hn/content/división-de-gobierno-digital

51 Access to this system is presently restricted to only the SCGG and the President’s Office.

52 The central education administrative system (Sistema de Administración de Centros Educativos, SACE).

53 In contrast, no CSO respondents working in education mentioned accessing or using the SACE data. This may be because the database is not entirely publicly available. Development partners seem to have better access, given their connections with the secretariat and government.

54 The Category 5 storm destroyed an estimated 70-80% of the transportation infrastructure, 70% of all crops, and over 30,000 homes. With 5,000 accounted deaths and 12,000 injuries, the storm caused $3 billion USD in losses (IDB, 2000).

55 This figure was calculated from the 2015 Honduras budget document published by SEFIN (the Secretary of Finance).

56 The five largest development partners in Honduras are the World Bank, the US, the European Union, Germany, and Canada (OECD, 2014).

57 Studies show that hiring international experts — a common practice among donors — can undervalue the importance of local knowledge (Rose et al., 2016).

58 The OECD (2009) notes that the HDI has disaggregated data by departments and municipalities and allows a mapping of the disparities within the country. It has been used as an advocacy tool by many local governments to draw the attention of central government when allocating public investment funds, aid funds or domestic funds intended to alleviate poverty.

59 Examples of sector-specific work were in malnutrition, child labor, women’s rights, agriculture, the labor market and climate change.

60 Another similar example was of an audit conducted by the Tribunal Superior de Cuentas with support of the World Bank in 2008 that found acute irregularities in teacher salary payments, with more than 10,000 teachers receiving additional payment benefits without meeting the necessary requirements for these rewards. This information was published in a report, and was used by this CSO to put pressure on the authorities to take action.

61 Smaller working groups within the MERECE are attended by development partners with a specific focus within education.

62 While the larger CSOs tend to develop their strategies and plans independently, smaller CSOs often form alliances and submit joint project proposals to compete for funding. Many are also part of a federation, FOPRIDEH, which serves as a platform for meeting and sharing information. Several respondents use FOPRIDEH’s database, which contains reports, publications and other type of analyses aimed at monitoring and influencing public policies.

63 For instance, one respondent mentioned using a GPS tagging-enabled tool for this.

64 This assessment rubric was introduced in the study From Pork to Performance: Open Government and Program Performance Tracking in the Philippines by Custer et al. (2016). The 4C’s framework draws upon Tiago Peixoto’s “minimal chain of events” for an accountability mechanism built on disclosure principles from The Uncertain Relationship Between Open Data and Accountability: A Response to Yu and Robinson’s “The New Ambiguity of ‘Open Government’” (2013).

65 For example, it was reported that municipal officials tended not to collaborate with enumerators, but were more likely to cooperate and share information if they belonged to the same party as the central government.

66 While these data may be collected in traditional formats, they may not always be consolidated and published as a database.

67 Municipalities keep track of their spending and revenue regardless of whether they are using SAMI, though the latter can help reduce information asymmetries between the central and local governments. Each quarter, municipalities must submit audit reports to the Department of Municipal Strengthening to be eligible for a transfer from the central government. In one case, the mayors alerted the secretariat of education (SoEd) of out-migration by youth in their municipalities, due to a lack of technical studies or professional middle education. The SoEd responded by opening 41 new professional centers.

68 Since the law mandates reporting foreign aid information to the SIAFI, some development partners report to the SIAFI only and not to the PGC, while others assume that the data can be pulled from SIAFI. The secretariat for external relations (SRECI) has recently begun working with SIAFI to register donors; however, it is difficult to manually crosswalk the information in the PGC and the SIAFI. Staff from SIAFI and SRECI meet twice a month to reconcile the information, as the same donor may report different amounts in the two platforms.

69 Each project and each CSO has a different methodology for data gathering and evaluation, which makes it difficult for other projects and CSOs to use these data as a base. For instance, a CSO participant mentioned that for food security, it is important to know the date at which the indicator was measured, since data may vary from one season to the other. In the absence of such information, it becomes difficult to use data from other stakeholders. Donors and consultants also bring new indicators, which results in data sets that are difficult to compare over time.

70 For example, interviewees identified such a reorganization in 2014 as displacing experienced staff trained on the use of the aid information management system (PGC), which, in turn, contributed to lower levels of government capacity to use the PGC data.

71 For example, a lack of GPS capabilities at the regional level makes it difficult to track climate information that farmers need to plant. Technicians are reportedly underpaid and lack mobile phones, which makes it difficult for them to solve technical problems in the field during a drought.

72 An example that was mentioned in an interview was of the secretariat of education and the secretariat of infrastructure having complementary data, but not sharing it.

73 Some line ministries can access parts of the integrated financial management system (SIAFI). For example, the secretariat of education can access some information on education.

74 The integrated financial management system (SIAFI) is a database with comprehensive budget information that is not available to the public; however, the government publishes PDF reports that are publicly available. Even government representatives cited limited access to the SIAFI data. Another example is the CENISS database, which has individual-level social and economic data that allows the Office of the President to identify the beneficiaries of social programs and subsidies. However, this information is only available to the President’s Office.

75 For instance, since 2006, access restrictions to the national institute of statistics’ (INE) household survey data have increased. These data used to be available for $10,000, but since 2014 they cannot be accessed at all (interview).

76 Interviewees cited two development partner databases collecting information on the security sector, neither of which is publicly available.

77 Even though we captured the views of some technical junior-level staff, our target respondents were fairly high-level decision-makers and influencers. It is therefore possible that awareness of the AIMS is higher among technical staff that manages databases and data systems.

78 A few respondents who mentioned using the PGC indicated that it was not up-to-date, and may not have complete information in terms of capturing flows to all development partners and civil society groups.

79 One poignant example raised during the interviews was the fact that one development partner project supported both genetically modified and organic agriculture in the same locality.

80 It should be noted that the Law of Official Secrets came into effect in March 2014, which has limited the access to public information, as it protects several government entities from releasing information.

81 Use of the PGC appears to be limited to the SRECI, but respondents acknowledged the benefit of broadening the user base.

82 Several respondents indicated that current platforms are disconnected, and there is a need to integrate information from different institutions, so they can better inform decision-making. The SGPR is an initiative in this direction.

83 The SDP has four pillars: Social Capital, Infrastructure, Economic Development, and Institutional Framework.

84 Host of the g7+ secretariat, Timor-Leste endorsed the New Deal for Engagement on Fragile States that includes three sets of development principles (g7+, 2016). Timor-Leste has improved on measures of peace and security, but remains prone to small-scale conflict and access to justice is variable (Fragility Assessment Team at Ministry of Finance, 2013; UNDP, 2012).

85 The accountability chain includes various parts of the development process: external money (e.g., Extractive Industries Transparency Initiative, the Aid Transparency Portal); expenditures (e.g., Budget Transparency Portal, eProcurement Portal); and results (e.g., Government Results Portal). See: https://www.globalpolicy.org/component/content/article/165/29635.html

86 We identified health and agriculture as the focus sectors for Timor-Leste based on aid data quality (coverage, precision), aid portfolio prominence (projects, value), and policy priority, including alignment with the SDP.

87 Our population of interest includes development stakeholders that either make or inform decisions regarding the financing, allocation or targeting of development activities in Timor-Leste. In February 2016, we held interviews with 29 government officials, 23 development partner representatives, and 25 CSO representatives, to produce a sample of interviews that was representative of the broader population. While the findings are broadly representative of development stakeholders based in Dili, the capital, they do not speak to the views of officials and staff that work at the subnational level (departments and municipalities).

88 The budget process includes four stages: preparation, consideration, approval, and execution.

89 The Ministry of Finance works closely with each line ministry to prepare their budget submission which is reviewed by the Council of Ministers to define an annual budgetary envelope. The Ministry of Finance then prepares a budget circular, “which defines fiscal ceilings for each agency, category and major program” (La’o Hamutuk, 2015). These determinations are made based upon available budgetary resources (calculated based upon domestic revenues data from the Central Bank, past expenditures, and budget execution numbers from previous 2-3 years from the government’s budgeting system); available aid resources (calculated based upon the government’s Aid Transparency Portal); and strategic priorities (based upon the SDP and line ministry sector strategies). Following review by the Budget Review Committee and approval by the Council of Ministers, the draft budget law is presented to Parliament with detailed “budget books” (La’o Hamutuk, 2015).

90 Several CSO respondents noted that opportunities that had previously existed for CSOs to participate in the budget process (eg. through presentations to parliament) had been closed off in FY2015 and FY2016.

91 This system, developed by Freebalance, includes information on obligation, expenditure, and distribution of funds; references have been made to monthly, quarterly, and semi-annual reports.

92 For example, in the agriculture sector working group, the Ministry of Agriculture and Fisheries requests development partners to report on relevant activities, including: project title, geographic coverage, total budget, annual budget, start year, end year, duration (in years), implementing agency, donor, and project description.

93 Initially funded through support from the Asian Development Bank, JICA, and AusAID, the ATP was implemented by AidData Center for Development Policy (ACDP) partner Development Gateway via their Aid Management Program (AMP) beginning in July 2011. The public portal can be accessed at aidtransparency.gov.tl. In 2014, the Development Partnership Management Unit (DPMU) launched a public aid map portal with assistance from ACDP and funding from the USAID Global Development Lab, which made reliable information on subnational aid project locations available to the public for the first time. Ongoing maintenance for the Aid Transparency Portal is funded by Timor-Leste’s general budget.

94 Timor-Leste was the third country in the world to be awarded EITI compliance status. “Government Results Portal Launched”, Press Release, The Secretary of State for the Council of Ministers and Official Spokesperson for the Government of Timor-Leste, April 24, 2012. See: http://timor-leste.gov.tl/?p=6852&lang=en

95 “2010 Revenue Watch Index”, Revenue Watch Institute and Transparency International. See: http://www.revenuewatch.org/rwindex2010/pdf/RevenueWatchIndex_2010.pdf

96 “The Statistical Work Plan for Timor-Leste, 2010-2019,” National Statistics Directorate, Ministry of Finance. See: http://www.paris21.org/sites/default/files/TIMORLESTE_SWP2010-2019.pdf

97 The government enacted the country’s first formal press law, the Media Act in November 2014. However, many have criticized the law for providing insufficient protections for a free media. The law generally acknowledges a right to information; however, Freedom House (2015) finds that this right to information is toothless, as “the law offers no details on government obligations or enforcement” and “is often arbitrarily blocked by the government.”

98 One respondent noted that cocktail parties are more effective than official channels to gain access to data.

99 One respondent noted that gaining access to one government data system requires minister-level approval.

100 According to UNDP (2012), Timor-Leste is the second most oil-dependent nation in the world. Also see: http://www.laohamutuk.org/Oil/EITI/2012/SSCOMResultsPortal24Apr12En.pdf

101 Some interview participants noted that government interest in information on ODA activities has increased recently, due to declining oil prices that are making development cooperation a more prominent part of Timor-Leste’s budget.

102 According to the 2014 Development Cooperation Report produced by the Development Partnership Management Unit, official development assistance (ODA) delivered in 2014 as off-budget grants was $262.7 million or approximately 10% of the Combined Sources Budget for Timor-Leste, which includes both domestic resources through the general state budget and ODA. The Government of Timor-Leste defines off-budget aid as external assistance not included in the state budget. See Annex 2 for historical ODA commitments. Sources: aidtransparency.gov.tl; Budget Book 1, https://www.mof.gov.tl/wp-content/uploads/2013/10/FINAL_20131024_Budget_Book_1_EN_16.10.pdf. In FY2016, the government cut funding to key development sectors (agriculture by 18%, education by 2.7%, and health by 37.2%) in favor of increased funding for large-scale infrastructure projects. This budget was vetoed by the President, but the veto was unanimously overridden by Parliament (Ruak, 2015; La’o Hamutuk, 2015).

103 Broadly speaking, development partners can be divided into two types: those whose strategic plans are largely defined by headquarters (HQ-dominant) and those who develop country-specific strategic plans with HQ playing a largely supporting role (HQ-autonomous). For both development partner types, a funding envelope is approved at headquarters or home government level.

104 The guidelines, reporting frequencies, and results frameworks for implementing partners may be determined during project design, specified in the terms of reference, or delineated after the awarding of a project.

105 For more information, see: http://dfat.gov.au/about-us/business-opportunities/tenders/Documents/timor-leste-m-e-house-final-design.pdf

106 For more information, see: http://asiafoundation.org/resources/pdfs/VisitorSurvey2014English.pdf

107 One CSO participant described the challenges with CSOs working at cross-purposes, including an instance in which one CSO paid farmers to attend trainings, limiting the ability of other CSOs who did not offer remuneration to secure attendance.

108 This assessment rubric was introduced in the study From Pork to Performance: Open Government and Program Performance Tracking in the Philippines by Custer et al. (2016). The 4C’s framework draws upon Tiago Peixoto’s “minimal chain of events” for an accountability mechanism built upon disclosure principles from The Uncertain Relationship Between Open Data and Accountability: A Response to Yu and Robinson’s “The New Ambiguity of ‘Open Government’” (2013).

109 For example, the agricultural extension program established in 2008 (Ministry of Agriculture and Fisheries, 2008) is responsible for data collection for the SLMS, but suffers from a lack of trained personnel and limited government support (Lundhal, 2013). Many respondents cited funding shortages (e.g., for bicycles and extension worker per diems) and bureaucratic constraints (e.g., traveling to Dili to obtain fuel vouchers), which have hamstrung the effectiveness of these local-level sectoral data collection systems. Respondents also noted that extension workers and health clinic officials may not be motivated to report accurate data in the absence of a feedback mechanism for them to understand how the data is used and the importance of data quality.

110 One CSO respondent specifically noted that the Ministry of Agriculture and Fisheries’ ability to improve the effectiveness of its agricultural extension program is stymied by a small budget, which is largely allocated toward staff salaries. This constraint is exacerbated by FY16 budget cuts to these sectors.

111 As one development partner expressed, since they lack their own internal system to systematically track project locations to the district level, they suspected that some of the location information for their programs in the Aid Transparency Portal was also likely incorrect.

112 For example, here is a sample of government census and survey reports, with their most recent year in parentheses: National Census (2010); Household Income and Expenditure Survey (2011); Timor-Leste Survey of Living Standards (2007); Labor Force Survey (2013); Demographic and Health Survey (2009-10); and Business Activity Survey (2013). In addition, several of these reports are not available in Tetum, a widely spoken official language of Timor-Leste, on the National Statistics Directorate Website.

113 The Development Partnership Management Unit asks development partners to update their Aid Transparency Portal data every three months; however, compliance is a challenge.

114 In the context of this study, we use the term “network” to refer to one’s circle of professional and personal contacts or acquaintances from which one may source information.

115 For example, one implementing partner that was unaware of the Aid Transparency Portal expressed demand for a system much like it and noted that the Ministry of Agriculture and Fisheries was hoping to create a database to track aid projects in the agriculture sector.

116 Per the World Bank (2014) definition, Internet users are individuals who have used the Internet (from any location) in the last 12 months. Internet can be used via a computer, mobile phone, personal digital assistant, games machine, digital TV, etc. Internet access was stunted from 2002-2013 due to the monopoly power of Timor Telecom that reduced competition and kept prices prohibitively expensive. While new firms have entered the market since 2013, Internet access remains out of reach for many and inconsistent for those who have it (World Bank, 2012).

117 While respondents noted that Aid Transparency Portal (ATP) data is likely more complete than the data collected via this sector-specific system, to use ATP data they must phone a contact in the Ministry of Finance to access the system and print the requested information, due to poor Internet connectivity. Respondents instead prefer to use their sector-specific data collection system, because they can receive the information directly in an Excel template shared via email. If they have questions, they can call the point of contact at each development partner.

118 For example, according to UNICEF (2013), Timor-Leste ranks 134th out of 186 nations in adult literacy at 58.3%.

119 While the full National Census report is not available in Tetum, the suco-level census profiles are published in Tetum.

120 Some respondents noted that, due to low numeracy and statistical literacy, data visualizations can also be difficult to interpret. They advocated for data visualizations in context to promote understanding. Budget allocation visualizations, by La’o Hamutuk, an independent Timor-Leste NGO, were cited as one such example. See: http://www.laohamutuk.org/econ/OGE16/WholeBudgetCashJan16en.jpg

121 One development partner noted they were conducting 3-4 different household surveys to collect slightly different data from the same population.

122 Respondents noted that the Ministry of Agriculture and Fisheries and the Ministry of Health were restructured 3 times since 2012, suggesting that while the restructuring was implemented with the intent to improve government performance and efficiency, these gains generally have not been realized.

123 All line ministry respondents identified that the ATP is not sufficiently granular for line ministries to identify the proportion of on-budget funding that would flow to them, rather than funding development partner and implementing partner salaries and overheads. This was identified as a barrier to ATP use and effective budget planning.

124 Related work by Development Gateway to assess the uptake of results data among policymakers and practitioners provides some good insights on building up data leaders: http://www.developmentgateway.org/assets/post-resources/RDI-PolicyBrief.pdf

125 Since the completion of this research, the Development Partnership Management Unit made the Aid Transparency Portal available in Tetum and upgraded the software for better functionality in low-bandwidth environments.

126 Editor’s note: Due to the specific areas of expertise of those interviewed in Senegal, this chapter’s recommendations are aimed specifically at increasing uptake of data from the aid information management system (AIMS) portal to advance Senegal’s broader national development strategy. Nonetheless, the findings from this study hold broader implications for development data that extend beyond aid information alone.

127 Senegal underwent a peaceful democratic election in 2012 and held a constitutional referendum in March 2016 that placed term limits on the presidency and strengthened the rights of opposition parties.

128 The Sustainable Development Goals (SDGs) substantially shaped Senegal’s three-pillar Emerging Senegal Plan and a five-year Priority Action Plan (AllAfrica, 2015). The Government of Senegal was also an early champion of the Global Partnership for Sustainable Development Data, a multi-stakeholder initiative of 150+ governments and organizations working to fill data gaps for the SDGs. http://data4sdgs.org.

129 Health and education were selected as the pilot sectors for this initiative.

130 To limit the scope of the study, we identified education and health as the focus sectors for Senegal based on a variety of factors such as: data quality (coverage, precision), aid portfolio prominence (projects, value), and policy priority, including alignment with the ESP. However, we expect these sectors will provide insights on the opportunities to promote data uptake across all sectors.

131 Our population of interest includes development stakeholders that either make or inform decisions regarding the financing, allocation or targeting of development activities in Senegal. We sought to produce a sample of interviews that was representative of the broader population. While the findings are broadly representative of development stakeholders based in Dakar, they do not speak to the views of officials and staff that work at the subnational level (departments and municipalities).

132 Eighteen months prior to the start of a new fiscal year, the Ministry of Finance creates the Table of Financial Operations, which calculates the amount allocated to each Ministry based on the government’s revenue, International Monetary Fund (IMF) mandates, and other political commitments. Twelve months prior to the new fiscal year, each line ministry submits their Politique Sectorielle (annual strategy), outlining their priorities for the year. The DCEF uses these inputs to draft the AIP seven months prior to the new fiscal year.

133 CONGAD is a consortium of NGOs and civil society actors that assist the Ministry of Health, development partners, and other stakeholders in the planning, implementation, monitoring, and evaluation of the national strategy.

134 The AMP was developed and supported by AidData Center for Development Policy consortium partner, Development Gateway.

135 From 2000-2015, these three development partners contributed more than $80 million each to the health sector. The contributions of the top 5 largest development partners in health are available in Appendix C.

136 From 2000-2015, France contributed nearly $1 billion to the education sector, four times that of the second largest development partner, Canada. The contributions of the top 5 largest development partners in education are available in Appendix C.

137 The G-50 has a rotating presidency with one multilateral and one bilateral serving as co-presidents on two-year terms.

138 The UN and Spain currently serve as co-presidents of the G-50. The co-presidents administer G-50 meetings, coordinate directly with the President and Prime Minister of Senegal, and relay the strategic interests of the development partner community to the Government of Senegal.

139 Based upon annual financial commitments.

140 The World Health Organization (WHO) and Government of Canada serve as the chairs of the health and education sector working groups, respectively.

141 This assessment rubric was introduced in the study From Pork to Performance: Open Government and Program Performance Tracking in the Philippines by Custer et al. (2016). The four C’s framework draws upon Tiago Peixoto’s “minimal chain of events” for an accountability mechanism built on disclosure principles from The Uncertain Relationship Between Open Data and Accountability: A Response to Yu and Robinson’s “The New Ambiguity of ‘Open Government’” (2013).

142 For example, although the District Health Information System (DHIS) has significantly improved data quality and management compared to its predecessors, Ministry of Health officials indicated a need for actors at the ‘local infrastructure’ level (i.e., hospitals and clinics) to enter data into the DHIS instead of rolling it up to the district level, thus facilitating localized data management.

143 There are conflicting impressions among interviewees about the reason for the delay, as either being due to the Division of Economic and Financial Cooperation’s concerns about the accuracy of the AMP data or stemming from a desire to maintain control over the information. Regardless of the motivation, the lack of a public portal has lowered the AMP’s visibility among government stakeholders and hampered the ability of development partners and CSOs to interact with it and leverage it regularly.

144 Political issues also arise in the way that other line ministries perceive the Division of Economic and Financial Cooperation (DCEF), and DCEF perceives itself, “above the rest” as the final arbiter on the budget. This factor contributes to political tensions and prevents broader awareness, sharing, and use of AMP data across the government as DCEF has chosen to retain sole production rights on AMP data.

145 While Development Gateway helped build the system and serves in an ongoing advisory role, the Senegalese government owns, operates, and makes all decisions regarding the maintenance of the platform and the data within it.

146 Subsequent to our field research in the first quarter of 2016 and in dialogue with our partner Development Gateway, the Ministry of Finance has made important strides in line with our recommendations, including: increasing staff allocated to the DECF, and using data from the International Aid Transparency Initiative (IATI) to augment Senegal’s AMP data.

147 For example, the Global Partnership for Sustainable Development Data has held a series of national workshops to help countries articulate priority data needs to advance monitoring and action for the SDGs and identify partners to help fill gaps. The PARIS21 Secretariat has also played an important role in helping countries conduct needs assessments for SDG monitoring and budgets for national statistics systems (PARIS21, 2016).

148 Custer et al (2012) adapted these archetypes to the discussion of open data from Eric Raymond’s original use of the terms in his 1999 book, The Cathedral and the Bazaar, to explain different approaches to software engineering.

149 In the world of development finance, for example, the OECD Creditor Reporting System offers a high degree of accuracy for those seeking to conduct rigorous time series analysis and who do not mind a time lag of two years. Meanwhile, the International Aid Transparency Initiative offers more timely information for forward planning purposes, but the quality of the information is more variable across donors.

150 It is important to note that the utility of these minimal quality assurance standards and processes is in their consistent application in practice, rather than their existence on paper.

151 Of course, there are limitations of the wisdom of the crowd. The risk of “participant fatigue” is high if citizens see little return on their investment of time and effort (Gigler et al., 2014). Citizen-generated data can also come with its own errors that could compound inaccuracies (Custer et al., 2012).

152 When the Government of Honduras had to curb the country’s deficit in 2009, it chose to cut the budget of its national statistics agency (Instituto Nacional de Estadística, INE) to reduce costs. However, this also increased the dependence of the INE on external financing to create and maintain databases that it needed to produce statistics for donors and line ministries.

153 The International Open Data Charter is a global, multi-stakeholder action network that offers six principles as a common foundation for governments to “realise the full potential of open data for their own jurisdiction.” The first principle is that [government] data should be “open by default.” See: http://opendatacharter.net/principles/

154 Governments are often the major focus of pushes for greater transparency, but CSOs and development partners should not be forgotten, as they also produce information that could be of high value, but is often not publicly available.

155 AidData will publish the results from the field experiment in sometime in 2017.

156 Paris21 does offer a good primer on lessons learned in “How to Build Statistical Capacity,” which includes a distillation of tactics they have found to be effective in practice, though this is based upon qualitative observations. See: https://www.paris21.org/sites/default/files/Capacity-buiding-knowledge-series.pdf

157 The training will include how to access the aid information management system, search for data, generate reports from the platform, export data, and use the system to support a range of development planning activities. In addition, the Honduras experiment will assess the effectiveness of different informational appeals to prospective participants to attend the training event. AidData will publish the results in 2017.

158 Brian Levy coined the original phrase in his 2014 book, Working With the Grain: Integrating Governance and Growth in Development Strategies. Custer et al. (2016) adapted this concept for open government and technology-enabled transparency.

159 Zeitlin et al. (2014) chronicle the origins of positive deviance in studies of child malnutrition dating back to the 1960s as a concept to describe why some children appeared to grow and develop adequately in spite of living in impoverished environments.