Avoiding Data Graveyards: Insights from Data Producers & Consumers

Section 2.2.1.1
Granularity: Government, civil society, and development partner leaders want more data that is disaggregated by sector, geography, and demography

There was a groundswell of interest across government, development partner, and CSO interviewees in having more data on development infrastructure (e.g., clinics, schools) and results (e.g., student performance, unemployment figures) that is disaggregated by district (or relevant subnational unit). Yet, demand for this information outstrips the capacity of the three countries to produce it consistently. When asked about high-value data sources, interviewees most frequently mentioned geo-referenced and sector-specific administrative data produced by line ministries, and surveys and censuses from national statistical offices.28

In Timor-Leste, for example, respondents expressed interest in greater disaggregation of survey and census data by district, subdistrict, and village (three administrative levels).29 In Senegal, development partners specifically requested disaggregated information on crosscutting issues such as gender parity, climate change, and governance, as well as other sector-specific indicators. In Honduras, interviewees across stakeholder groups reported interest in municipal-level socio-economic data and sector-specific indicators, such as: out-of-school children, topographic conditions for crops, and school-based violence.

This interest in disaggregated information is consistent with reports from other countries and international experts. Analyzing preliminary usage patterns from Kenya’s Open Data portal, Rahemtulla et al. (2011) identify a high degree of citizen interest in more granular or localized information. USAID (2015), in a series of Aid Transparency Country Pilots, highlights the lack of data disaggregation as a specific challenge and reports demand among those interviewed for more subnational information. The UN (2015, p.11) notes in its reflection on the Millennium Development Goals (MDGs) that having more, high-quality data disaggregated across various dimensions (e.g., age, sex, ethnicity, disability, geography) will be “key to making decisions and monitoring progress in achieving sustainable development for all.”

If disaggregated data is in short supply relative to demand, this is a missed opportunity to encourage broader use of government data. This supply-side failure is a likely by-product of the fact that the production of highly granular sector-specific data relies on providers at the point of service delivery (e.g., school administrators, medical professionals, agricultural extension workers). Front-line workers often have constrained data skills to report information accurately, and limited visibility on its value if they do. This dynamic creates a stunted feedback loop with weak incentives for front-line providers to change the status quo (Custer et al., 2016a).30

Section 2.2.1.2
Accuracy: Development data sources are frequently viewed as incomplete or out of date due to technical capacity constraints and perverse political incentives

Governments, particularly national statistics offices, are among the most prolific data producers in low- and middle-income countries. Unfortunately, concerns regarding the timeliness, accuracy, and validity of government-produced data substantially dampened broader use among those we interviewed. While most often associated with government data, these challenges extend beyond the public sector. For example, development partners often do not allocate sufficient attention to reporting on their activities in a timely manner, hence concerns raised in all three countries regarding the timeliness of data published via aid information management systems.

Interviewees cited several technical constraints that compound the risk of inaccurate data, including: publication delays, episodic data collection, limited data management capacity at the subnational level, and gaps in coverage. In Timor-Leste, front-line service providers are under-resourced, under-trained, and under-motivated to submit complete and accurate reports. In Honduras, financial and technical constraints were cited as the primary reasons why data reported by local staff is inaccurate. In Senegal, school and clinic-level data is still collected via pen and paper before digitization at the district level.

While data producers grapple with substantial technical capacity gaps, interviewees additionally emphasized that there are also political reasons driving inaccuracies in development data. Interviewees noted that powerful incentives exist for public servants to make the official numbers look good.31

Inaccuracies in official data sources, whether intentional or unintentional, are not unique to Honduras, Senegal, and Timor-Leste, however. Devarajan (2011, para 8) acknowledges that “weak capacity in countries to collect, manage, and disseminate data” is a contributing factor, but points to politics as the underlying driver to explain the poor state of statistics in Africa. Election cycles create domestic pressures for politicians to falsify statistics to garner the support of constituents (Devarajan, 2011). Development partners can also inadvertently create perverse incentives for officials or front-line providers to misreport on their country’s progress through tying performance incentives to meeting certain targets as reported via administrative data the government collects (Sandefur and Glassman, 2014).32

Unfortunately, as most government ministries have low data management capacity (particularly when it comes to procedures for data verification and auditing), there are limited internal safeguards to mitigate this risk of data manipulation, which further lowers user confidence. In response, development partner and CSO respondents resort to fielding independent data collection efforts to supplement or circumvent the use of official data. These unofficial data collection efforts are a missed opportunity to validate and augment official data sources. As Sandefur and Glassman (2014, p.24) argue, “getting administrative data and [independent] survey data to speak to each other” can create powerful checks and balances to improve accuracy.

Section 2.2.1.3
Silos: Development decision-makers need to triangulate diverse data points, but disconnected data initiatives make integration and interoperability difficult

Whether they are making new investments, monitoring current progress or evaluating past performance, decision-makers need to see the full service delivery chain from upstream budgets and intermediate outputs to downstream outcomes. Unfortunately, fragmentation is often the norm in Honduras, Senegal, and Timor-Leste, as it is in many other parts of the world: disconnected data points are siloed between disparate information management systems maintained by different agencies and collected using non-standardized methods (Custer et al., 2016a). As a result, decision-makers and their support staff must expend significant time, money, and effort to put this information to practical use.

Having launched an ambitious national development strategy in 2015, the administration of Senegal’s President Macky Sall has since emphasized the role of data to guide sectoral priorities and budget allocations. The Honduran government, meanwhile, is developing a centralized results-based management system to integrate data across ministries on inputs, outputs, and outcomes. To succeed in these efforts, public sector leaders will need to address head-on the intractable issues of integration (data sharing) and interoperability (data harmonization).

These challenges extend beyond the government. Development partners and CSOs expend substantial resources to produce development data that meets their specific needs. These independent data collection exercises often use different methodologies and generate datasets that may not be publicly available, making comparisons difficult. In Timor-Leste, several interviewees noted the substantial risk of reinventing the wheel with ad hoc data collection efforts that are seldom coordinated with each other, even within departments of the same organization. One development partner reported that they were conducting 3-4 different household surveys to collect slightly different data from the same population.

Data silos not only are costly to maintain and frustrating to navigate, but are also viewed by several interviewees as likely to increase the risk of data gaps and errors.33 The United Nations (2015) shares this perspective, arguing that integration of data sources and adoption of standardized methods and formats benefits data producers and consumers alike by reducing the likelihood of errors, long-term costs, and respondent burden (e.g., participant fatigue or partial compliance in response to multiple data requests).

In the face of consequential capacity gaps, governments, development partners, and CSOs could enjoy gains from a more concerted effort to make their data sources comparable across sectors, geographies, and time periods, as well as through shared training and technical staff. However, it is clear that connecting information islands will require political leadership to convince competing agencies and independent organizations that the tangible benefits of adopting common data formats, re-engineering data collection processes, and changing institutional arrangements outweigh the perceived pain points (United Nations, 2015).

Section 2.2.2.1
Access: Absent Freedom of Information legislation, development data access depends upon commitments to voluntary disclosure

The rise of information communication technologies is decreasing the cost, time, and effort to produce a wealth of development data; however, greater availability of information does not necessarily translate into equitable access. In Timor-Leste and Senegal, both countries without Freedom of Information (FOI) laws, visibility on public sector activities depends on the government’s willingness to open up this data, or on the strength of one’s network to secure it via back channels. This reliance on voluntary disclosure creates uneven access that disproportionately affects less well-connected groups and individuals. In Senegal, civil society groups cannot readily access the national statistics agency’s data and civil servants struggle to obtain hyper-local, disaggregated data managed by other ministries.

Even in countries that have adopted FOI legislation, access to development data is only as strong as the government’s commitment to uniformly enforce compliance. In Honduras, much of the sector-specific indicator data in demand is either not available or not open, despite high-profile transparency commitments.34 In the agriculture sector, the government recently launched an internal government platform for monitoring agricultural outcomes, which is not public. The national statistics agency charges a fee for some of its household survey data and the Ministry of Education’s database of education indicators35 is not yet entirely publicly available.

Development partners and CSOs are also often reluctant to share information and they typically do not fall under the jurisdiction of FOI law. For example, interviewees report that development partners and CSOs conduct independent data collection exercises and maintain proprietary databases that have valuable information on development projects and indicators, but are not in the public domain. Some central government officials, that view development partners and CSOs as bypassing them to collect information directly from local government officials without sharing their results, also hold this perception.

Unfortunately, this mismatch between supply and demand is not unique to the three countries in this study. When governments exempt certain categories of information or types of agencies, they dilute the power of FOI laws. Meanwhile, voluntary disclosure practices, such as open data initiatives, are often the product of compromise (Gigler et al., 2011; Custer et al., 2016a). The datasets that governments and organizations make available are the easiest to publish, not the ones most in demand (Davies, 2014). A government or organization often chooses which data to make available in isolation, rather than through a process of gauging public interest in various data (Canares and Shekhar, 2015).

In fairness, government agencies, development partners, and CSOs often face dueling imperatives: paying for large-scale data collection and management infrastructure using their own resources, while still answering the call to make data freely accessible and open to all. Advocates for transparency and open data have arguably been more successful in generating political pressure for openness than in helping governments and organizations grapple with the sticky issues of business models to cover the costs of data collection and management sustainably. In addition, as Custer et al. (2016a) describe in an assessment of the Philippines’ open government initiatives, data producers are also concerned about non-monetary costs, such as citizens misinterpreting information or accusing them of misleading the public if data is inaccurate.

Section 2.2.2.2
Connectivity: Delivering development data exclusively via online channels makes access difficult due to connectivity and human capacity constraints

The proliferation of smart phones and increasing Internet penetration in the past decade has done much to democratize data and reduce information asymmetries. However, there remains a profound divide between those that are part of this digital revolution and those left behind (World Bank, 2016; Gigler et al., 2014). Limited Internet connectivity, language barriers, and low levels of data literacy and numeracy inhibit practical access to, and use of, development data in many countries and stakeholder groups.

The case of post-conflict Timor-Leste is particularly poignant. In multilingual Timor-Leste, development data is often available online in the official languages of Portuguese and/or English, but not in Tetum, the lingua franca of most citizens.36 In addition, a weak educational system has led to a dearth of professionals trained in literacy, numeracy, and computer skills. As a result, domestic policymakers and practitioners are predisposed to favor an oral culture of information sharing and are far less comfortable using complex online databases.

Even among the 1.1% of Timor-Leste’s citizens that use the Internet (World Bank, 2012), interviewees report that slow and intermittent connectivity makes it difficult to share information via online channels. For example, one line ministry reportedly developed a parallel system to collect data on development cooperation activities via Excel spreadsheets emailed to users that lack a stable Internet connection. Other respondents resort to calling contacts to access online systems on their behalf, or using the in-person development partner working group meetings as a workaround to get the information they need.

Connectivity and capacity constraints in utilizing online sources of development data are not unique to Timor-Leste. If government, development partner, and CSO producers of development data rely exclusively on online dissemination strategies, this can inadvertently privilege access to information among the urban, educated, and well-to-do (who tend to be more digitally savvy) at the expense of other groups (Read and Atinc, 2017; Custer et al., 2016a).

Section 2.2.2.3
Invisibility: Prospective data users have low familiarity with many publicly available information sources

Even when data is available and accessible, it may not be visible enough to prospective users for them to take advantage of this information in their work. Nonetheless, government, development partner, and CSO producers seldom have an intentional strategy to proactively communicate and promote the development data they publish via datasets, databases, and data portals. For example, development partner representatives interviewed in Honduras, and even fewer CSOs, were aware of publically available sources of development cooperation data, such as the country’s open aid information management system. People will not use development data that they do not know exists, increasing the risk of data graveyards.

A similar narrative emerges from other data initiatives beyond this study. In an assessment of governance data use in 126 low- and middle-income countries, Masaki et al. (2016, p.3) note that producers rarely have a “proactive strategy to conduct outreach with domestic media outlets” even though media is “one of the top ways users say they learn about new governance data sources.” Similarly, Custer et al. (2016a) find that in opening up information on public service delivery, the Government of the Philippines and the World Bank put substantial time and effort into connecting disparate data islands, but paid comparatively little attention to promoting the existence of the public data portals. Advertising was often limited to a one-off launch event and/or ad hoc training events with local journalists.

Section 2.2.3.1
Credibility: Government-produced data suffers from a trust deficit, with many respondents viewing these data sources as out-of-date and unreliable

As is often the case in low- and middle-income countries, government-produced data in Honduras appears to suffer from a trust deficit, which undercuts use. CSO representatives in particular raised concerns that mayors and other officials sometimes skew data to receive more funding (e.g., adjusting population figures) or under-report bad news to paint a rosier picture than reality. Interviewees raised examples such as: teachers abetting students to cheat to boost test scores, nurses not registering people with Chikungunya (a mosquito-borne illness) to lower the disease incidence rates, and the lack of sound population data making it difficult to calculate the percentage of children in school.

Section 2.2.3.2
Incentives: Data is only one part of the calculus, and leaders may still ignore evidence in favor of convenience or organizational imperatives

As development policymakers and practitioners make decisions about which programs and projects to fund, they inevitably create both winners and losers. In this respect, resource allocation decisions are likely fraught with more political jockeying than the design or evaluation of the resulting projects. This is consistent with the experience of interviewees across the three countries who report that upstream budgeting and planning processes are substantially shaped by existing strategies, organizational imperatives (e.g., headquarters’ directives, tied funding), or political exigencies (e.g., protecting interests of allies, alleviating concerns of vocal constituencies).

Some government, development partner, and CSO representatives across the countries reported using data in the context of siting new projects (e.g., feasibility studies) or assessing project results at closure. However, even at these parts of the project cycle, development partners and CSOs make decisions from more than the evidence alone, as they may select project locations based on ease of access, available infrastructure, and existing networks. Political pressures or the desire to maintain good working relationships can also undercut willingness to frankly discuss (or publicize) lackluster results in implementation, thereby impeding organizational learning and corrective action.

Beyond organizational factors, societal norms and individual biases can also decrease the perceived value of drawing upon data and evidence throughout policymaking processes or project cycles. In Timor-Leste, for example, interviewees emphasized the country’s strong oral culture and indicated that government officials often have an unfavorable view towards using data and evidence because they associate this information with international experts.

When government, development partner, and CSO representatives choose to use data and evidence in their decision-making, they pay the costs upfront in time, reputational risk, or lost political capital for uncertain future rewards. A Development Gateway (2016, p.3) assessment of results data use emphasizes that there is seldom an explicit requirement that development decision-makers in low- and middle-income countries base their decisions upon evidence, and few rewards if they do so. In fact, data analysis and evidence-informed policymaking is viewed as “an overtime activity…[only] for the intrinsically motivated.”

The absence of clear rewards for making decisions based upon evidence, and of penalties for not doing so, gives tacit permission for policymakers and practitioners in all three countries to pay lip service to the importance of development data, while allowing other considerations to crowd out using this information to change the ways decisions are made.41 It is telling that the most frequent use case mentioned by interviewees was data for reporting purposes, namely as inputs to report on progress to development partners or the Chief Executive’s office. It remains to be seen whether results-based management and evidence-informed policymaking initiatives, such as Senegal’s effort to emphasize a greater role for data to guide sector priorities and budget allocations in the education and health sectors, will fundamentally alter the status quo.

Section 3.1.5.1
Development partners produce data and evidence that is broadly perceived to be reliable and useful

Development partners create databases and specialized reports, fund data collection by the government, build capacity through initiatives to strengthen data systems, and publish information about their interventions in evaluation reports. Most do not collect detailed project-level data on monitoring themselves, working instead through their implementing partners (e.g., CSOs or contractors). However, development partners often hire foreign consultants to undertake these activities rather than working through the government or local CSOs, which effectively deters building a strong cadre of domestic data scientists and statistics professionals.57

Nonetheless, development partners produce indicator data that all stakeholder groups find to be reliable and useful. One frequently cited example was the United Nations Development Programme’s Human Development Index (HDI). The secretariat of external relations uses the HDI in its aid information management system (PGC), and development partners leverage the data source to perform country assessments. Meanwhile, CSOs find the HDI helpful to justify interventions in the least developed municipalities of the country.58 Interviewees also mentioned macro-statistics from the International Monetary Fund and poverty maps produced by the World Bank and the Inter-American Development Bank as data sources that were useful in their work.

Development partners publish sector-specific analytical reports to inform strategic planning efforts and evaluate in-country work.59 CSOs also identified development partner-produced analytical reports as particularly useful to help them advocate with the government for policy changes or develop stronger systems. For example, one CSO cited a World Bank report on money lost to absentee teachers as their inspiration to create a system to verify teacher attendance and advocate with the government to be more conscientious in tracking this information.60

Section 3.1.5.2
Information sharing is key to coordination, but happens behind closed doors

The government has an important role to play in curbing fragmentation and ensuring alignment of development partners with national development priorities. In 2014, more than 30 development partners were operating in Honduras, most contributing a small amount of funds to the country (OECD, 2016). Larger agencies do attempt to coordinate with each other, but there are perverse incentives among development partners and NGOs to advance their own agendas at the expense of coordination. Development partners do report using the G-16 group of the 16 biggest donors in Honduras as a forum to share information with each other.

Typically, development partners create broad multi-year and country-specific strategic plans that outline their geographic and sectoral focus areas. Often, development partners time these country plans to coincide with a presidential election, and ensure alignment with the incoming administration’s priorities and new Strategic Plan. Government and development partner collaboration varies by sector, with interviewees reporting that they work together closely in the education sector.

For example, the International Cooperating Partners’ Education Round Table (Mesa Redonda de Cooperantes Externos en Educación, MERECE) serves as a forum for development partners to share information on their projects, and for the government to present its priorities and financing needs.61 The sector roundtables – mesas sectoriales, tripartite bodies that are part of the G-16 and comprise government, CSO, and development partner representatives – are theoretically another venue for dialogue. However, their success depends upon the organizing development partner’s ability to ensure participation and continuity between administrations.

While development partners produce formal sources of data and evidence, they also rely heavily on informal back channels to get information and insights from each other on development cooperation activities. The sector roundtables allow development partners to exchange information on sector-specific projects and locations with their counterparts. Nonetheless, these venues have varying levels of participation and success, incentivizing development partners to turn to their personal connections and networks to access information on a just-in-time basis. One possible interpretation of this: development partners may view their personal connections or professional networks as complementing publicly available data sources, providing additional context in the form of lessons learned and experiences working with certain partners or communities.

Standardization is also a problem of disrupting the status quo and the country politics. People use the data to their advantage and those who control it like the power that it gives them. All these things considered, the government wants to improve on this [but] it is difficult and costly to do so.

Section 3.1.6.1
Civil society organizations are an untapped resource for highly granular project-level data

CSO-produced development data could complement official data, especially at the local level. A critical part of the local data ecosystem, CSOs generally produce three types of information: baseline social data from the communities where they work, monitoring and evaluation data from the projects they implement, and reports by consultants on specific subjects. CSOs collect hyper-local data that could be a highly valuable source of information to help fill persistent gaps in official data collected at the subnational level. Yet, broadening the availability and use of CSO-produced data may be impeded by the fact that these data collection exercises often begin as a workaround. In the absence of reliable, comprehensive, and granular data from government channels, CSOs seek out alternatives to capture their own disaggregated data on development resources, activities, and outcomes. This project-level data production can be costly and time intensive, particularly for smaller organizations.

Section 3.1.6.2
Civil society organizations come in different shapes and sizes, but offer similar examples of data use

CSOs use development data in a variety of ways to inform their strategies, plans, and activities. For example, CSOs often use baseline socio-economic indicator data produced by development partners to select project locations or monitor changes over time and report on results. Large CSOs have more established ways to coordinate and share information when targeting communities to ensure no duplication of efforts.63 Smaller organizations, in comparison, typically rely on phone calls and on-site visits. CSOs also make use of thematic meetings and conferences organized by development partners to interact and exchange information with others in their sector.

[Our organization] aligns with the national plan of Honduras, because donors align with the national plan. Getting money from donors is easier if you have similar goals to the national plan.

Section 3.2.1.1
Granularity: Disaggregated and geo-referenced data is in high demand, but low supply

There is strong demand across stakeholder groups for socio-economic data and sector-specific indicators at the municipal-level, such as: out-of-school children, topographic conditions for crops, and school-based violence. The National Statistics Institute (INE) was frequently mentioned as an important source of this disaggregated data, which is often used as a base to construct other indicators. However, there was an expressed desire on the part of government, CSO, and development partner representatives alike for the INE to produce more up-to-date, accurate, and publicly accessible data.

For example, census and household survey data collected by the INE is obsolete – the last agricultural census was in the 1990s – and lacking in coverage for certain groups (e.g., indigenous populations). In addition to financing constraints, gang violence and political opposition makes the work of enumerators challenging in some municipalities.65 Government ministries and development partners sometimes undertake supplemental data collection efforts on an ad hoc basis for specific needs; however, this is insufficient to provide a complete, comparable subnational dataset.66

Information costs a lot to generate, and the ones that generate it don’t want to share.

A federal initiative, the integrated system of municipal administration (Sistema de Administración Municipal Integrado, SAMI), endeavors to capture municipal-level budget, tax, and expenditure data but includes information for only 40 out of 298 municipalities, as of May 2016.67 Municipalities also report on development projects and outcomes in their jurisdiction; however, national-level officials sometimes perceive this locally reported data to be inadequate or inaccurate. This disconnect may partly be attributed to the fact that the local government officials collecting downstream results data are less directly involved in upstream allocation and planning decisions and have little insight into how this information is used (Development Gateway, 2016).

Section 3.2.1.2
Silos: Disconnected data initiatives undercut the accuracy and utility of development data, and increase the costs of reporting and maintenance

Stakeholders frequently identified difficulties in dealing with fragmentation: disconnected data points, silos between disparate information management systems, and maintenance by different agencies. This status quo hampers the ability of even government officials to share information between ministries. Several databases are only accessible to the ministry owner, and there is no centralized system to access different government data sources.

Yet, integration and standardization are perceived to be prohibitively difficult and expensive. This creates perverse incentives for ministries to duplicate efforts and capture similar information. For example, the Government of Honduras maintains two systems that contain information on foreign aid — a financial management system used for primarily on-budget projects (SIAFI) and an aid information management system for planning and coordination (PGC). These systems may serve different functions, but sharing data could create effiency gains.68

Disconnected systems also impede the ability to leverage complementary data from multiple agencies. A new results-based management system (SGPR) being piloted by the government is an improvement over the status quo, as it will pull together data from all existing government databases. However, this is currently envisioned to be accessible only to the Office of the President with the focus on monitoring indicators in the national plan. However, the SGPR could be a unique opportunity for the government to share this information and connect the dots along the entire service delivery chain, from data on resources (e.g., domestic budgets, grants, external loans) to outcomes (e.g., socio-economic outcomes, project results).

These challenges also extend beyond the government. Poor coordination between government agencies, development partners, and CSOs leads to inefficiencies in data collection. Indicators are measured using different methodologies, a particular challenge in using baseline data.69 Datasets are not easily comparable over time, as development partners often bring in different consultants to work on the same subject over the years, each with their own ideas about how to measure development indicators. This problem is accentuated by staff turnover within development partner offices, which often means that there is little institutional memory.

Section 3.2.1.3
Capacity: Human, technical, and financial constraints perpetuate inefficient data supply that is out of step with user demands

The electoral cycle has an unintended consequence on the government’s capacity to produce development data: a loss of trained personnel and the resources spent on training them to collect, curate, and maintain data. As new administrations take office every four years, each comes with its own priorities. The resulting reorganizations can displace experienced staff to other government offices or out of government entirely.70 These capacity gaps are interlinked with financial and technological constraints.

External resources still largely fund national data capacity, creating islands of excellence in line with development partner priorities. As a result, some government institutions have better data systems than others, and the consistency of data collection varies. For example, the National Statistics Institute (INE) did not conduct a national census after 2001 or an agricultural census since 1993, partly due to lack of funds. The 2012 census was financed by the Inter-American Development Bank (IDB). Almost all respondents in the agricultural sector mentioned the lack of a recent census as a problem. Lacking another source of information in the agriculture sector, a CSO participant mentioned using the 1993 agricultural census for their current analysis.

Meanwhile, in the education sector, teachers in rural areas have limited Internet access, for example, and need to travel to towns to be able to enter the data into centralized systems. Government respondents in the agriculture sector highlighted a lack of funding, technical personnel, and technology and new techniques for recording more precise data.71

Section 3.2.2.1
Access: Absent a culture of information sharing, data is often available, but not accessible

Government ministries or units that collect raw data do not always share it, or there are delays in putting these data into the public domain.72 A few government respondents acknowledged that there was no culture of sharing information, and that information from other ministries would be very useful in their work.73 While the government may release PDF reports, many of the underlying databases it maintains are classified, which impedes broader use of these data among the public.74 Despite the existence of high profile government transparency initiatives, several interviewees have actually observed a decrease in their access to government data in recent years, as information is hidden behind either paywalls or firewalls.75

There is also a perception that development partners do not always make public the results or outcomes of their projects. Several government and civil society group respondents reported that development partners sometimes bypass the central government to approach municipalities directly for information, or support closed databases that collect similar or duplicate data.76 This challenge was not limited to data on project inputs, but also included information on lessons learned and project outcomes, which development partners do not always make public.

Civil society organization participants expressed a need for more information on the results and impact of projects to enable learning. This was not specific to projects implemented by CSOs, but a perceived need in general. It was mentioned that when projects fail, the reaction is often to hide the data, which does not allow for other actors to learn from those mistakes. This is a missed opportunity for other parts of the government or non-governmental actors to access the same set of information and iterate together.

In the case of non-reimbursable funds, [development partners] don’t feel the commitment to report back to the government. If the donor wants to hide, they hide.

Section 3.2.2.2
Familiarity: Even when data is accessible, it may not be visible enough to prospective users

People cannot use development data that they do not know exists. Certainly, this is true for proprietary databases and reports that are hidden from public view, but this can also affect open data portals if they are not proactively communicated and promoted amongst prospective user groups. Honduras’ aid information management system (PGC) is case in point: despite being open to the public, only a third of the government and development partner representatives interviewed (and even fewer CSOs) were even aware of its existence. 77

Duplication of efforts in development cooperation (and in data collection specifically) was cited as a concern across stakeholder groups. Government officials spoke of poor coordination with development partners, and between development partners and mayors of municipalities; development partner and civil society organization officials lamented poor coordination among development partners. Most respondents expressed a desire to know what others are doing, particularly in their sectors of interest, and the need for greater synchronization of efforts (and funding) among all these actors.

Strikingly, these same groups reported limited awareness of the aid information management system (PGC), which has publicly accessible project-level information that can be used to facilitate coordination within and between government agencies and development partners on development assistance activities.78 Unfortunately, the lack of visibility of open data sources like the PGC makes it difficult to stoke demand from domestic constituents, which dampens interest in addressing supply-side constraints to produce timely and comprehensive development data.

Section 3.2.3.1
Credibility: Government-produced data is plentiful, but there is a trust deficit with prospective users of this information who doubt its veracity

As is often the case in low- and middle-income countries, government-produced data in Honduras appears to suffer from a trust deficit, which undercuts use. CSO representatives in particular raised concerns that mayors and other officials sometimes skew data to receive more funding (e.g., adjusting population figures) or under-report bad news to paint a rosier picture than reality. Interviewees raised examples such as: teachers abetting students to cheat to boost test scores, nurses not registering people with Chikungunya (a mosquito-borne illness) to lower the disease incidence rates, and the lack of sound population data making it difficult to calculate the percentage of children in school.

Section 3.2.3.2
Incentives: Data is only one part of the calculus, and leaders may still ignore evidence in favor of convenience or organizational imperatives

Even when government, development partner, and CSO leaders have access to credible and relevant development data, there are limits to what this information can do. Individual incentives, long-standing country strategies, and organizational imperatives often compel decisions to be made based on convenience or intuition, rather than evidence. The challenge of curbing aid fragmentation and improving coordination is particularly illustrative.

I don’t [put much] trust in government data. When I go to the field the reality is always different.

Development partner and government representatives frequently express the need for better coordination, but there is little indication that they use data to allocate resources and plan aid activities. For example, one respondent cited a sector roundtable (mesa sectorial) that collected information from development partners on the amount and location of investments, and identified areas of duplication. However, no real measures were taken to improve aid coordination in the sector. Development partners still fund contradictory activities in the same communities or duplicate efforts despite the available evidence.79

People want to use their common sense and not the data…They think that the data is telling them things that they already know, and so they don’t think it is important

Section 4.1.1.1
National budget planning and implementation is an important entry point for data on development resources and outcomes

The Ministry of Finance monitors budget implementation through an expenditure review analysis unit which tracks the status of projects based upon reports generated from its Performance Management System.91 Line ministries develop their own information management systems to collect project implementation and results data (e.g., the Ministry of Agriculture and Fisheries’ Suco-Level Monitoring System and the Ministry of Health’s Health Management Information System).

Line ministries also collect information on the sector-specific activities of development partners to support budgeting, inform planning, and monitor performance. In some cases, they employ their own reporting templates to obtain information via sector-specific working groups (e.g., agriculture and health).92 In addition, development partners report their activities to Timor Leste’s aid information management system, the public Aid Transparency Portal (ATP) managed by the Ministry of Finance’s Development Partnership Management Unit.93 The ATP replaced a previous Excel-based data collection method, and is the most comprehensive source of aid information in Timor-Leste. Table 1 provides a description of various sources of aid information available in Timor-Leste.

Ministry officials reported using data not only to plan, execute, and monitor their own programs, but also to inform their interactions with other parts of the government, foreign governments, and development partners. For example, Ministry of Finance respondents noted that they are requested to pull information from the Aid Transparency Portal to provide reference information for the Ministry of Foreign Affairs in advance of Policy Dialogue meetings with external partners that provide foreign assistance.

Section 4.1.1.2
The government voluntarily discloses information via open data initiatives; however, these transparency gains are not durable absent a Freedom of Information law

Under the Gusmāo administration, Timor-Leste launched several open government portals to disclose information, including: the Budget Transparency Portal, the eProcurement Portal, the Aid Transparency Portal, and the Government Results Portal. It also established itself as a global leader in transparency: joining the Extractive Industries Transparency Initiative (EITI) in July 201094 and receiving a ranking of “comprehensive revenue transparency” in the 2010 Revenue Watch Index.95 These efforts on the part of the government were bolstered by investments from the World Bank, the IMF, and others to build the country’s national statistical capacity.96

However, Timor-Leste lacks a formal Freedom of Information (FOI) law and has only weak protections for the media (Freedom House, 2015).97 CSOs go through lengthy and difficult formal request processes or rely on their personal and professional networks to access data.98 Therefore, while the government has, laudably, chosen to make some data public via its Government Transparency Portal, accessing other high-value government data sources is difficult.

In the absence of clear guidance for what data should be made publicly available, many government officials default to keeping data private until given explicit permission to share. There is a strong “culture of deference and respect for hierarchy” that extends to data disclosure practices – senior-level government approval is often required to access or release information (Freedom House, 2015).99

Section 4.1.2.1
Development partners are power users of development data, but strategic imperatives diminish the role of objective evidence in decision-making

Although development partners vary in their division of labor, typically several actors inform the decision-making process – each serving distinct, but complementary, roles. Headquarters may provide strategic guidance, typically in the form of broad goals to inform country strategies, or they may take a more “hands-on” approach to developing or approving strategies to be implemented by Country Offices at the national level.103 Within Country Offices, Program Officers oversee operational or technical areas of a development partner’s portfolio and manage the activities of implementing partners, while Technical/Support Offices serve crosscutting functions across projects (e.g., finance, monitoring and evaluation, etc.).

Development partners reported using data on development outcomes, past project performance, and the activities of other development actors to: formulate strategic plans, assess needs, design projects, and establish baseline outcome measures to monitor progress. Development partners often seek to leverage existing data, such as government statistics (e.g., census) or third party data (e.g., Demographic and Health Surveys or Asia Foundation data) for these purposes. However, many respondents reported that they were unable to find existing data that was fit-for-purpose. In these instances, development partners fund primary data collection efforts to be implemented internally or by an external party.

Yet, interest in data does not always translate seamlessly into decisions. Development partner decisions may be driven by strategic imperatives from headquarters rather than available evidence, reducing the motivation of development partner officials to use data. Since most development partners implement programs through contracted services, their implementing partners make consequential decisions about the design, implementation, and monitoring and evaluation (M&E) of projects. Data used by development partners is most likely to impact these decisions when written into the terms of reference or directly passed along to their implementing partners.104

Some development partner respondents are evaluating how to collect M&E data from implementing partners that can most effectively be used as a decisional input by development partner staff. For example, the Australian Department of Foreign Affairs and Trade is launching a new M&E program that will institute an integrated performance management system focused on increasing uptake of M&E data in decision-making.105

Section 4.1.2.2
Development partners are interested in using development cooperation data to curb fragmentation and improve targeting, but are frustrated by duplicate reporting systems

Given the number of development partners operating in Timor-Leste, coordination is a strong use case for official development assistance (ODA) information to avoid duplication of efforts. Most development partners expressed confidence in their ability to obtain a complete picture of relevant development cooperation activities during project design through extensive, in-person stakeholder consultations. However, others recognized the potential for a skewed picture that excludes smaller actors or those that weren’t available to attend a meeting during the consultation period.

In addition to personal networks, many development partners reported using Timor-Leste’s public Aid Transparency Portal (ATP) to identify the right individuals to speak with for stakeholder consultations and select project locations. Development partner respondents also raised other use cases for the ATP to inform more effective targeting of their assistance to districts with the greatest need and learn from experience to improve project design and avoid repeating mistakes. However, some development partners expressed frustration with the duplicate reporting requests they receive to report into the ATP and via the sector-specific schemes.

Subnational information, improving development partner coordination, and ensuring “no one is left behind” may go hand-in-hand. For example, when collaborating with the Ministry of Agriculture and Fisheries to design a new agriculture program, one development partner reported that they used the aid map in the ATP to identify potential gaps in coverage and discovered that most of the proposed project locations were in districts in the western part of the country, which was already receiving comparatively more agriculture funding than the east. Armed with this information, the development partner insisted that the Ministry of Agriculture and Fisheries include some eastern districts in the program.

Section 4.1.3.1
Civil society organizations may fund their own data collection efforts to fill perceived gaps

As CSOs implement projects and conduct primary research, they produce and collect data that can be of broader use to other development practitioners and policymakers. In some cases, CSOs self-fund data collection efforts to fill perceived data gaps, even when external resources are not available. For example, when confronted with a lack of tourism data, the Asia Foundation piloted an effort to compile this information by surveying individuals entering Timor-Leste at the airport, which facilitated a full-scale study.106 However, the self-funding model is only feasible for those organizations with the resources – both time and money – for the effort.

Section 4.1.3.1
Civil society organizations highly prize disaggregated and geo-referenced data at the sub-district level

CSOs reportedly consider several factors in determining whether to undertake new activities, including: alignment with organizational strategy, availability of funding, and their geographic presence. Funder priorities may influence the selection of target districts and activities, but CSOs typically determine the specific project locations (i.e., villages or communities). Therefore, it is perhaps unsurprising that CSO respondents expressed greater demand for spatially precise data on development indicators at the sub-district level than other respondents. Precise information on official development assistance-funded activities at the sub-district level is also important to CSOs that need to coordinate with other organizations working in the same geographic area or want to identify prospective funders of their work that are active in their sectors of expertise.107

Table 2 provides a recap of the main decisions made, data types used, and unmet data needs of each of the major stakeholder types included in our study.

Section 4.2.1.1
Accuracy: Development data that is incomplete or out of date raises skepticism among prospective users regarding the reliability of available information sources

Accuracy of government data sources was frequently cited as a concern, exacerbated by substantial resource and capacity constraints. Sectoral data systems, such as Ministry of Agriculture and Fisheries’ Suco-Level Monitoring System (SLMS) and the Ministry of Health’s Health Management Information System (HMIS), were identified as particularly unreliable. Many noted that the local-level officials responsible for reporting information into these systems are under-resourced, under-trained, and under-motivated, yielding incomplete and inaccurate reporting.109

As a result, these government data systems can produce wildly inaccurate statistics. One respondent reported a 30-40% gap between immunization coverage reported in the HMIS versus an independent survey, while another suggested that the SLMS consistently overestimates agricultural yields. These challenges are exacerbated by the recent budget cuts to key social sectors, reducing the ability of line ministries to collect and use high quality data.110

Development partner respondents expressed concerns that the public aid information management system (ATP) does not provide accurate information on the subnational locations of their activities.111 One development partner specifically noted that they would be more likely to use the ATP data to design projects or coordinate with others if they had greater confidence in the quality assurance performed on the data and the resulting data quality.

I don’t trust the information on a website because I don’t know how recent or thorough it is.

Timeliness was also raised as a challenge for data sources, given the costs involved in fielding data collection efforts and the difficulty of securing sustainable funding. While the frequency of the government’s major surveys and censuses are in line with typical best practices, several respondents expressed concerns that these data are out of date.112 Similarly, many development partners cited lack of timeliness as the reason they prefer to go to their personal networks for up-to-date information on development cooperation activities, rather than use data maintained by the government.113 However, timeliness concerns extend beyond government data. A CSO respondent echoed this sentiment for third-party data sources in Timor-Leste, noting, “I don’t trust the information on a website because I don’t know how recent or thorough it is.”

Timeliness constraints compound accuracy concerns, as the government creates projections based on the 2010 census to extrapolate more recent statistics (e.g., population). Though such projections are common practice, respondents expressed concern that these projections may not be accurate due to shifting dynamics, and have concerns about the accuracy of the original survey, with one government representative referring to such statistics as “estimates on estimates.”

Section 4.2.1.2
Granularity: Despite a “wealth of available data,” there is still a persistent call for greater disaggregation and sector-specific statistics

Respondents were mixed on the issue of data coverage. Some take an optimistic view: citing the “wealth of data” available in Timor-Leste or the fact that the country is “over-studied.” Others were more pessimistic, especially in the agriculture sector. Several respondents noted the lack of available government data on agriculture and called for an increase in sector-specific statistics in agriculture and for the completion of the country’s first agriculture census planned for 2018. One government respondent lamented, “In agriculture, 99% of data is incorrect or incomplete.”

Beyond sectoral gaps, other respondents pointed to insufficient disaggregation in government data and advocated for access to raw budget and census data, as well as suco-level (village-level) disaggregation of government surveys. Central government officials expressed limited interest in the subnational location of development cooperation activities; however, many respondents indicated that this information might be of greater interest to local government officials or development partners.

In agriculture, 99% of data is incorrect or incomplete.

Section 4.2.2.1
Access: Absent Freedom of Information legislation or broad-based Internet usage, access to government data depends upon one’s network114

The view that government information was difficult to access was not universally held. Typically, less well-connected junior staff reported greater difficulty obtaining data directly and instead turn to open fora (e.g., social media and listservs) to obtain information. This is problematic for data-driven decision-making, as it is often these junior staff that are responsible for preparing analysis for their senior leaders to review. Respondents in senior positions had a more sanguine view, noting that government data was comparatively more accessible in Timor-Leste than other countries in the region, and that they could easily gain access to government data via informal channels.

While personal and professional networks in Timor-Leste can facilitate information sharing, in some cases lack of awareness of available data sources inhibits uptake. For example, we found that many respondents were either completely unaware of the existence of Timor-Leste’s public Aid Transparency Portal (ATP) or did not realize that they could access the data.115 As a result, the ATP is largely seen as a government reporting tool rather than a useful data resource.

There is also a pronounced digital divide: only 1.1% of the population uses the Internet and those who do lament that it is slow and unreliable (World Bank, 2014).116 Twenty-nine percent of respondents cited slow or no Internet access as hindering their use of data.

For example, one line ministry was forced to develop a parallel system to collect data on development cooperation programs directly from development partners due to declining Internet connectivity that made it impossible to effectively use the online ATP.117

This also extends to those contributing data via web-based reporting interfaces, as well. Some development partners noted that it could take 1-2 hours to report data into the ATP, while others had to travel to use computers made available by the government at the Ministry of Finance. These challenges can cause some development partners to report data to the ATP late or not report at all, compounding the trust concerns discussed earlier.

Section 4.2.2.2
Capacity: Low levels of data literacy and numeracy, as well as language barriers, inhibit the effective use of development data in Timor-Leste

A weak educational system perpetuates low levels of human capacity in Timor-Leste (UNICEF, 2013).118 Few adults have had mathematics training and literacy and numeracy levels are low; however, fixing the education system is perceived to be a lesser priority relative to other goals (Anis, 2007; Marx and Pinhero, 2013). Meanwhile, the highest capacity individuals are often recruited to work for development partners who can offer much higher salaries, leaving a shortage of skilled labor which “adversely influences the effective functioning of the government” (UNDP, 2012).

As a result, the government’s capacity to use data for decision-making is limited, with many respondents indicating that long-form reports and dense statistics simply do not resonate. Some respondents noted that the government has limited capacity and resources to step up to expectations that it leads sector strategic planning, coordinate the efforts of development partners, or make evidence-informed policy changes. However, others cited improvements in Timor-Leste’s education system following independence, and expect that this will bring more highly educated individuals into government and increase data-driven decision-making in the future.

As in other countries, senior leaders and mid-level technocrats often disagree on their preferred forms to receive data. Senior leaders prefer data to be interpreted for them and reported in polished analysis products. Meanwhile, mid-level technocrats often prefer to conduct their own analysis using raw data. These distinct use cases are often difficult for any single data product or portal to accommodate. For example, senior decision-makers expressed frustration with the complexity of the public Aid Transparency Portal, preferring to navigate quickly to top-line numbers. In contrast, mid-level technocrats felt that navigating through charts, maps, and reports slowed their ability to access and export raw data.

Language is another critical barrier for many Timorese, with information frequently published in English or Portuguese rather than the far more commonly spoken Tetum. Indeed, only two of six major National Statistics Directorate surveys are publicly available in Tetum.119 Several respondents emphasized that in the oral culture of Timor-Leste, easily understandable data visualizations120 and in-person outreach are important ways to circumvent these language barriers and to convey complex information more simply.

Section 4.2.3.1
Credibility: Many development partner and CSO respondents view government data sources as lacking credibility and field their own independent primary data collection efforts

When asked about their use of official government data sources, 48% of development partner respondents and 38% of CSO respondents said they avoid or limit use of government data due to a lack of trust in the data sources. This pervasive distrust appears to stem from three underlying factors: lack of timeliness, lack of accuracy, and lack of coverage. Questions about methodological rigor and accuracy in how data was collected, curated, and published were also raised as challenges to data use.

For example, concerns were raised about whether the public aid information management system (the Aid Transparency Portal, or ATP) provides a complete picture of development cooperation. CSO activities are likely underreported, since these organizations do not currently report their programs in the ATP. Development partners may report on those CSO activities they fund; however, this misses information on projects funded by private donors or development organizations outside of the country. Similarly, development partners noted that the ATP may not capture all aid modalities (e.g., technical assistance, bilateral support to multilaterals). One development partner respondent argued that because the Ministry of Finance has prioritized the collection of financial data on aid programs, this comes at the expense of fully reporting other pieces of information, such as detailed project descriptions or project results.

In light of this trust deficit, many development partners and CSOs field independent primary data collection efforts to supplement or circumvent their need to use government data. However, several respondents noted the high risk of “reinventing the wheel,” as these ad hoc data collection efforts are seldom coordinated with each other, compounding challenges of duplication, and inefficiency. Organizations that engage in data collection, or even different departments within the same agency, may have perverse incentives that mitigate against coordination, such as securing contracts or budgets for themselves.121 When duplicative efforts are identified, it is often too late in the process to make course corrections, given reluctance to share early-stage plans.

Lack of confidence in available data comes at a substantial cost: duplicative data collection has created a patchwork landscape of disconnected datasets that are not easily comparable due to varying methodologies and scopes, and yield conflicting results. As they sought to reconcile data sources to create a summary agriculture report, one government respondent reported turning up a seemingly never-ending set of conflicting surveys.

Section 4.2.3.2
Incentives: Powerful external and internal pressures dilute the influence of development data in making evidence-informed decisions

Even when issues of trust, access, and capacity are overcome, there are no guarantees that development data will be used. Headquarters’ directives, tied funding, and alignment with long-term strategies may undercut the ability of development partners and CSOs to make data-driven decisions in their programming. Path dependence can lead development partners and CSOs to choose project locations based upon networks and existing infrastructure, rather than targeting project locations based upon need-based calculations or a desire to reduce duplication of efforts. Discomfort with delivering “bad news” can undercut frank disclosure of monitoring and evaluation results and hamper the ability to make course corrections during implementation. Professional and political incentives may lead decision-makers to ignore evidence in favor of a more expedient decision.

Meanwhile, many respondents perceive that senior-level government officials are predisposed to distrust data and not inclined to use it in decision-making due to their interactions with experts peddling advice ill-suited to the context in the immediate aftermath of independence. To counteract this backlash, several respondents emphasized the importance of cultivating strong interpersonal relationships based upon trust with Government of Timor-Leste units to promote data and foster policy change. They noted that this “quiet approach” through collaborative face-to-face conversations has yielded much greater success than arms-length data sharing or a confrontational approach. However, these important personal relationships are regularly disrupted by the frequent restructuring within the government of Timor-Leste.122

Section 5.1.1.1
10-year sector-specific strategies substantially shape not only planning and budgeting processes, but also the government’s data collection priorities

In the education sector, for example, the Ministry of Education establishes priorities aligned with its 10-year sector strategy and collects indicator data to track progress against these objectives. Functional directorates implement policies and programs for students in their assigned age-range and collect school-level indicator data for monitoring and evaluation. Regional-level indicators on school, teacher, and student performance that the ministry needs to track progress against the national education strategy remain the most sought after pieces of information.

The Directorate of Planning and Education Reform (La Direction de la Planification et de la Réforme de l’Education, DPRE) aggregates age-range and school-level indicator data from various functional directorates, as well as standardizing and verifying data received from development partners and CSOs. The DPRE uses this data to produce an annual statistical report on progress in the education sector. The Ministry of Education uses the DPRE annual report to communicate with development partners and CSOs to ensure strategic alignment of their activities with Ministry of Education priorities and advocate for funding on specific initiatives.

The Ministry of Education’s National Institute for Research and Action for the Development of Education (INEADE) also uses this information to conduct in-depth evaluations on desired educational outcomes metrics across the sector. The Ministry of Education draws upon these INEADE evaluations, the DPRE annual report, as well as other data from the National Institute of Statistics (NIS) to provide rationale for projects, monitor progress, and determine resource allocation for schools across the country. This, in turn, shapes the Ministry of Education’s annual strategy (Politique Sectorielle), which outlines its annual priorities and funding requests. Table 1 visualizes how government units work together to produce and use data to shape Senegal’s education policies.

In the health sector, Senegal’s national health strategy is primarily shaped by a steering committee comprising heads of various directorates within the Ministry of Health (MoH), the donor working group for the health sector, and the Council of Non-Government Development Support Organizations (Conseil Des Organisations Non Gouvernementales d’Appui Au Développement, CONGAD).133 Each year, these actors assess the strengths and weaknesses of the Ministry of Health’s strategy and make adjustments to better align performance with targets. These adjustments are reflected in the Ministry of Health’s annual sector strategy and inform coordination efforts.

Section 5.1.1.2
Sector-specific indicator and infrastructure data collected by the national statistics agency and line ministries are highly prized for decision-making

Sector-specific information on development indicators and infrastructure are highly prized and utilized in planning projects and evaluating progress in Senegal. In the context of this study, indicator data refers to the current status quo in a sector and given geographical area. For example, in the health sector, this might include: child mortality rates, the number vaccinated against polio, and the percentage with access to health care. Infrastructure data refers to the landscape of physical inputs designed to improve upon a given indicator within an area. For example, this might include: the number of clinics and hospitals by district, or the supply of medicines, doctors, and equipment by hospital.

Line ministries collect much of this sector-specific information. For example, the District Health Information System (DHIS) produced by the Ministry of Health is the backbone for data-driven decision-making in the health sector. It serves as the one-stop shop for providing discrete, relevant, and timely data on health indicators and infrastructure at all levels. Data in the DHIS is collected at each clinic or hospital, entered at the district level, and verified at the district and regional levels to ensure accuracy and timeliness.

The Ministry of Health produces an annual statistical report using sub-sectoral and geo-referenced health indicators to monitor the state of health in the country. Government officials also reportedly use geo-referenced health infrastructure data, alongside other data sources, to inform resource allocation decisions. Despite this interest in hyper-local data on health infrastructure, health officials did not strongly express a desire for other types of aggregate or subnational data (e.g., external or domestic financing for health programs).

In addition to line ministries, Senegal’s National Agency for Statistics and Demography (ANSD) is another important source of sector-specific indicator and infrastructure data. Under the auspices of Senegal’s five-year plan for statistical data collection and analysis activities — the National Strategic System — much of ANSD’s data is compiled via surveys fielded at various intervals from annually to once a decade.

The majority of ANSD surveys are administered via paper or field-based interviews that are recorded and later aggregated and analyzed. However, the ANSD is currently experimenting with a new method of survey data collection through the Listening to Senegal project, funded by the World Bank. In this pilot, the ANSD provides families with cell phones for their personal use and periodically calls these beneficiaries to ask targeted questions to collect information for their surveys. The ANSD makes recommendations based upon analysis of these surveys, such as advising government decision-makers on areas of greatest need.

Section 5.1.1.3
Development cooperation data may be an underutilized information resource among government officials who use it primarily to report to development partners

While official development assistance (ODA) is an important source of financing for activities in Senegal’s education and health sectors, the primary users of information on the distribution of these dollars and project results appear to be development partners, rather than the Government of Senegal. Government use of development cooperation data is limited to the Ministry of Finance’s production of its annual Official Development Assistance (ODA) report. While development partners acknowledge that the ODA report is useful as a top-line overview of development cooperation activities in Senegal, this information is not widely utilized by line ministries for sector-performance monitoring.

Ministry of Finance officials typically access data on ODA programs either through their professional networks or via Senegal’s Aid Management Platform (AMP). Hosted by the Division of Economic and Financial Cooperation (DCEF), the AMP is an aid information management system that tracks foreign assistance and public investment projects from the national budget.134 Two part-time DCEF employees currently maintain the AMP and request inputs from line ministries and development partners via an Excel template.

The Aid Management Platform templates contain “activity reports” listing all necessary fields and are populated using internal project documentation. These excel templates are not technically challenging to complete, but they do take time to fill out and the political incentive or mandate for doing so remains unclear absent high-level political will or demonstrated use cases. As a result, the platform is often incomplete (e.g., missing projects and fields) and out-of-date – as of late 2016, data on activities from 2014 had not yet been published – which limits its utility to a rough assessment of the aggregate contributions from each development partner in a given sector, as opposed to reliable estimates and disaggregated analysis.

Delays in publicizing data, lack of disaggregation, and low visibility of the platform beyond the Division of Economic and Financial Cooperation undercut its potential to play a broader role in helping line ministries, CSOs, and development partners to visualize and manage investments to achieve the Emerging Senegal Plan (ESP). Even though the platform may have most of the functionality needed to contribute in this way, the Ministry of Finance is undergoing a separate review of its technical infrastructure.

Section 5.1.2.4
Development partners are the primary users of development cooperation information, but they often bypass official systems in favor of informal networks

Development partners utilize a variety of data and evidence, including information on the activities of other donors, throughout the project cycle, conducting feasibility studies to plan and site new projects, as well as impact assessments at project closure. Detailed project documentation (e.g., scope of work, findings, baseline/endline studies) is of high interest to development partners seeking deeper insights on project design, results, and lessons learned. Development partners also expressed strong demand for disaggregated and geo-referenced information on official development assistance-financed (ODA) projects to determine where to site projects and avoid duplication of efforts.

Interestingly, in sourcing this development cooperation information, many development partners reportedly bypass official databases, such as Senegal’s aid information management system maintained by the Government of Senegal, in favor of obtaining ODA program data informally via sector working group relationships. This might involve development partners meeting with other organizations known to have run similar projects in the same region. Alternatively, development partners primarily use their own internal monitoring data and that of other development partners with whom they work closely.

While development partners regularly communicate with their government counterparts via several mechanisms, government interviewees noted that development partners do not always share data in a timely manner. Meanwhile, several development partner interviewees expressed the desire to see the Senegalese government take a more active leadership role to ensure the effective coordination of development financing and technical assistance. Nonetheless, the donor community in Senegal is divided on the value of investing in a more systematic system of information sharing to replace the status quo.

Section 5.2.1.1
Salience: There is a mismatch between the types of development cooperation data available and the information that is most in demand

Development partners view disaggregated information on development projects as useful to help them pinpoint the right actors to speak with for advice when planning to work in a specific sector or region. However, they were less certain regarding the usefulness of the type of information currently available in Senegal’s aid information management system (AMP). Rather than data on official development assistance (ODA) project locations, outputs, and finances – information published by the AMP – development partners instead expressed interest in qualitative information (e.g., evaluations and lessons learned), sector-specific indicator data disaggregated by geography, and data on crosscutting issues such as gender parity, climate change, and governance.

Section 5.2.1.2
Accuracy: Cumbersome data collection and reporting practices increase the risk of missing data and publication delays

Despite the government’s efforts to modernize the national statistical system in the past decade (Mauru, 2010), much of Senegal’s development data is still collected using antiquated methods. Primary data collection at the school or clinic level is often conducted using pen and paper due to limited Internet connectivity, computer access, and data management skills. This locally collected data is then sent via mail to be processed and digitized for inclusion in a digital database at the district or national level. Such a manual system compounds problems of timeliness, incomplete coverage, and poor disaggregation of data that, in turn, impedes effective use of that data for decision-making.

However, even when modern data collection methods are employed, cumbersome reporting practices undercut the reliability of official data sources such as Senegal’s aid information management system (AMP). Government officials and development partners alike view the AMP as less reliable, and a “second best” source of information to line ministry-curated databases or just-in-time data transfers between development partners.

Specifically, respondents voiced frustration with the AMP due to: publication delays (e.g., 2013 data on development projects was released two years behind schedule), missing fields, lack of standardization across donors, and duplicate reporting requirements (e.g., development partners being asked to report to both line ministries and the Division of Economic and Financial Cooperation (DCEF) for inclusion in the AMP).

Section 5.2.1.3
Capacity: Greater investment is still needed to professionalize Senegal’s national data and statistics infrastructure, particularly at the local level

Under the government’s leadership, Senegal has made important strides in increasing its technical capacity to produce high-quality statistical data, particularly with regard to poverty and education. However, substantial barriers remain for the government, development partners, and CSOs to effectively collect and use a broader suite of development data for decision-making in Senegal.

Funding and bandwidth constraints impinge upon the National Agency of Statistics and Demography’s (ANSD) ability to conduct timely national-level surveys. Implementation of the United Nation’s Post-2015 Development Agenda is likely to exacerbate this capacity gap, as there are greater demands placed upon the ANSD to assist with data collection to monitor and report on progress against the Sustainable Development Goals.

These technical and financial constraints also extend to line ministries. The Ministry of Education and the Ministry of Health struggle to recruit qualified statisticians in light of higher salaries available in the private sector. Meanwhile, within the Ministry of Finance, the lack of full time staff dedicated to maintaining the aid information management system hampers the DCEF’s ability to update the platform and ensure the integrity of the data, such as through conducting quality assurance checks against standardized reporting conventions.

Several interviewees from development partners and the central government also highlighted the need to improve data management capacity at the local level,142 including greater standardization in the way in which data is collected, curated, and aggregated. Data auditing functions are minimal, which increases the risk of data being manipulated to suit political needs or for data processing errors to create inaccuracies in the data. Another suggestion was to bolster funding for CSOs to assist the government to collect and verify hyper-local development data.

Section 5.2.2.1
Access: Data that is “out of sight” is also “out of mind” for prospective users

At present, data access relies upon the willingness of government agencies to voluntarily disclose information – something that they are not always incentivized to do. The absence of Freedom of Information legislation or executive branch transparency initiatives inhibits efforts to broaden the user base through increasing accessibility of these data. With official data sources locked behind pay walls and firewalls, the Government of Senegal is missing an opportunity to encourage data-driven decision-making within and outside of the government to advance the country’s national development strategy.

Civil society groups, for example, cannot readily access the National Agency for Statistics and Demography’s (ANSD) treasure trove of national statistical data. Even line ministries within the government have limited access to data disaggregated by location and demographics. Absent a high-level champion, Senegal’s aid information management system (AMP) – housed in a directorate-level agency three bureaucratic levels removed from the Prime Minister – was never publicly launched, despite political commitments to do so.143

Section 5.2.2.2
Networks: Satisfaction with sourcing development information via informal personal and professional networks depends upon who one knows

Development cooperation data makes for a poignant case study on norms and preferences of information sharing. While Senegal’s aid information management system (AMP) is not fully open to the public, development partners and government officials can still access the data. However, seven years after its initial implementation, many of these stakeholders did not even know that the AMP existed. Among those that were aware of the AMP, they were divided on the appeal of sourcing development cooperation data from a database versus seeking this information from their network connections.

In contrast, CSOs articulated the strongest demand for official development assistance (ODA) program data of any stakeholder group. This stakeholder group seldom is able to leverage the informal networks that development partners or government officials use to share information. Thus, CSOs would arguably have the most to gain from a publicly available, comprehensive, and up-to-date aid information management system. Several CSOs mentioned the value of this data to conduct feasibility studies to inform the siting of projects and resources. Others voiced an interest in producing independent analysis on development project investments and results for greater accountability.

Section 5.2.3.1
Incentives: Political factors often mitigate the ability of data to drive development decisions

Based on our interviews, it is clear that data is more commonly used at some parts of the decision-making process than others. Government officials and development partners report seldom using development data to set budgets and strategies. In this respect, development partner strategic imperatives and governmental politics appear to be more consequential than data in determining project placements or facilitating donor coordination. However, government officials and development partners are more likely to use data to plan and site new projects (as part of feasibility studies) or to assess results at project closure. This may imply that the distribution of resources is a far more politically contested exercise than project design and evaluation.144

Yet, even the use of statistical data can be derailed by politics. National Agency for Statistics and Demography (ANSD) analysis and resource allocation recommendations based upon census data are often outweighed by political calculations. The government’s aspiration to move towards making a greater number of decisions on data and evidence through its forthcoming National Statistical System may substantially alter this status quo; however, this will take not only greater data production capacity, but also a cultural shift in giving greater voice to data and evidence in critical resourcing decisions. Additionally, it was noted that resource allocation decisions are often made on a political basis, so the Government of Senegal or development partners do not always follow recommendations made by the ANSD.

Section 6.1.1.1
Principle #1: Producers should proactively engage citizens, companies, and civil society organizations as co-creators and validators of development data

Funders are rightly interested in strengthening domestic capacity to produce data. Advocates are motivated to push for more of this information to be open to the public. Governments and organizations may rush in to supply data in order to be responsive to citizens, national plans, or international commitments, but ultimately miss the mark if they fail to address concerns regarding granularity, accuracy, and gaps in coverage that undercut the use of that information.

The response to this quandary, Custer et al. (2012) argue, often pits two philosophies of data production against each other: “cathedral builders” versus “vendors at the bazaar”.148 In other words, should producers place the emphasis on taking time to perfect their data prior to release (the cathedral builders) or be quick to show their wares (the bazaar vendors), with the view that it is the responsibility of the “buyer” or consumer to beware of potential data inaccuracies?149

While data producers understandably face trade-offs in timeliness versus accuracy, the wisdom of the crowd, when harnessed properly, could enhance the cathedral. Beyond instituting a minimum level of quality assurance standards and procedures, the sole burden of verifying the accuracy of data need not fall on the producer alone.150 Government agencies, for example, could proactively enlist the support of citizens, civil society groups, and development partners to help catch errors in official data sources. For example, Custer et al. (2012) suggest crowdsourcing information from community members to remedy incorrect locations for local clinics and schools in Kenya.

There may also be an opportunity to crowdsource data from unofficial sources to augment or supplement the official record.151 Development partners produce data that is seen as reliable and credible by other stakeholders. Civil society organizations are a relatively untapped resource for project-level data about the communities in which they work. Individual citizens are well-positioned to speak to the quality of public services or local prices for basic goods and services. For instance, Custer et al. (2016) highlight a project of the Philippines Ministry of Finance, the World Bank, and the technology company Premise to mobilize Filipinos to report on local cigarette prices and monitor tax enforcement that generates revenues for public healthcare.

Closer partnerships between official and unofficial data producers could increase public awareness of government data sources and expand their coverage and completeness. Of course, this would require government producers to put a spotlight on potential limitations, gaps, and deficiencies in the data. Since the world of data is not immune to politics, this recommendation is likely most feasible in contexts where high-level government champions are able to create an authorizing environment for agencies to openly share data and collaborate with end users. Without this political cover, inertia will be difficult to overcome.

Section 6.1.1.2
Principle #2: Funders of data initiatives should align incentives of producers to favor integration and interoperability

The anticipated cost of producing data to support global goals and national priorities is substantial, and leaders in low- and middle-income countries already face a continual challenge to free up sufficient domestic resources to support their national statistics organizations and the data needs of line ministries.152 To get the most capacity for limited dollars, governments and their development partners should seize the opportunity to tie domestic and external financing for statistics and data with the expectation that producers demonstrate greater integration, interoperability, and openness in their data collection, management, and publication practices.

Funders of data and statistics initiatives should make it more difficult for government agencies, development partners, and civil society organizations to undertake duplicative data collection activities or maintain siloed data management systems. They should encourage both international and domestic actors to share information along the whole service delivery chain from upstream resourcing and infrastructure to downstream indicators and outcomes. Funders should also support the leadership of national statistics organizations to identify standardized indicators and definitions to guide measurements of development progress that are cross-walked to both country-level national development strategies and international frameworks like the Sustainable Development Goals (SDGs).

Devarajan (2011) gives a practical example of what creating the right incentives could look like, arguing that the World Bank and other development partners should ensure that they are funding data and statistics that are well aligned with national statistical development strategies. By implication, this requires funders to take the long view, eschewing one-off data extraction exercises that would allow them to quickly get data for their own purposes, and instead investing with an eye towards sustainability so that domestic actors have the capacity to produce and share data that meet their needs.

Badiee et al. (2016) propose the formation of data compacts that governments and development partners would use as a guiding framework to pool their resources, in response to a common set of statistical capacity priorities in line with a country’s national statistics development strategy. Meanwhile, in their assessment of African statistics systems, Sandefur and Glassman (2014) encourage domestic and external funders of data initiatives to explicitly invest in “getting administrative and survey data to speak to each other” to validate information and provide a more holistic (and accurate) picture of development progress (p. 24).

Section 6.1.1.3
Principle #3: Funders should invest in regular collection and proactive disclosure of geo-referenced survey and census data as a demand-driven priority

Domestic demand for disaggregated data frequently outstrips the capacity of national statistics organizations (NSOs) to supply this information. Funders wanting to make demand-driven investments in data and statistics should take note of this capacity gap and invest in the capacity for NSOs and other actors to conduct censuses and surveys in sectors such as education, health, and agriculture with greater frequency. Badiee et al. (2016) recommend an expanded program of censuses and surveys, which provides a good starting point for funders and official data producers to determine the optimal frequency of these data collection exercises (see Table 1).

Yet, these investments will generate real dividends only if producers geo-reference and disaggregate census and survey data by key demographic attributes (e.g., age, sex, disability status). As Sandefur and Glassman (2014) note, when household surveys provide only national estimates, they offer “little guidance to domestic policymakers [seeking to] allocate resources and attention between subnational units” (p. 6). Civil society organizations and local governments have the most to gain from access to precise subnational data, but often have the greatest difficulty in accessing this information.

Similarly, policymakers are hard pressed to ensure that no one is left behind if they are unable to monitor differential access to resources, services, and progress for vulnerable groups. As Badiee et al. (2016) argue, the implication of the Sustainable Development Goals (SDGs) is the need to disaggregate data by “the characteristics of people and their locations”, while still enabling aggregation and comparability across space and time (p. 13).

Funders and official data producers must also expand coverage of existing censuses and household surveys to include under-represented groups. According to Stuart et al. (2015), we have a “missing millions” problem: an estimated 350 million people worldwide are not currently covered by household surveys and, for all practical purposes, are invisible to those seeking to allocate resources and monitor progress against the SDGs.

Beyond the desire to be inclusive, ensuring that these “missing millions” are counted is imperative for three reasons: to accurately estimate whether the lives of the poorest of the poor are improving, to effectively target resources to those in greatest need, and to enable vulnerable groups to have a credible evidence base with which to advocate for better services and support.

Section 6.1.2.1
Principle #4: Producers should make data “open by default”,153 while funders and transparency advocates should work with them to make access sustainable

If funders want to get a better return on their investment and producers hope to increase uptake, both groups must proactively work together to broaden the reach of development data beyond their own organizations or agencies. They may find willing allies among open data hacktivists, open government reformers, and transparency activists.

There is growing recognition that national statistics should be open and that transparency initiatives should create norms and standards to ensure that people can easily access information they value. Data producers, both inside and outside of government, could benefit greatly from supporting broader transparency initiatives as a megaphone to promote visibility of, and access to, their data products. In return, transparency advocates may gain additional credibility and support for reforms if they are able to obtain the endorsement of agencies and organizations that are willing to open up their valuable development data.

Yet funders may have to break out of their technical or operational silos to exploit these opportunities to create an enabling environment for development data use. Funders of data and statistics capacity could make their investments contingent on producer willingness to adhere to open data principles and ensure sustainable access to that information through supporting FOI legislation.154 Meanwhile, funders of transparency and open data initiatives could incentivize producers to publicly disclose development datasets that are in highest demand and advise them on implementing open data principles in practice.

Producers have real concerns regarding the costs of data collection and the risks of increased public scrutiny. Funders could neutralize these deterrents to openness through helping producers develop sustainable business models and data management practices that cover costs and reduce their liability.

Section 6.1.2.2
Principle #5: Producers should partner with infomediaries – media, civil society, and think tanks – to promote data and translate it into actionable insights

Producers sometimes succumb to a narrow view of their mandate as limited to the collection, processing, and publication of raw data. To ensure their data products are being used, they must embrace a larger mandate for themselves as seeking “to translate raw [data] into actionable insights” (Custer et al., 2016, p.60). Producers also need an intentional dissemination strategy above and beyond publishing a dataset or holding a one-off launch event for a data portal. Otherwise, they are vulnerable to a ‘build it and they will come’ fallacy.

But data producers may not be well positioned to directly reach prospective target users outside of their organizations and agencies. Infomediaries (or information intermediaries) such as the media, community-based organizations, think tanks, and professional associations could play a complementary role in translating and communicating development data via various online and offline channels (Read and Atinc, 2017).

In brokering partnerships with such infomediaries, producers could adopt a hybrid dissemination strategy, as recommended by Gigler et al. (2014), to share development data that blends “high-tech” (e.g., Internet, social media, mobile phones) and “low-tech” (e.g., radio, print, community meetings) communications. This multi-channel approach is particularly important for data producers to overcome Internet connectivity constraints in low-bandwidth environments or in contexts where there is a low rate of Internet penetration.

Custer et al. (2016) spotlight an example of what these proactive producer-infomediary partnerships could look like in practice. Under the leadership of former Secretary Cesar Purisima, the Philippines Department of Finance (the data producer) collaborated with several media outlets (the infomediaries) to spark public interest in curbing tax evasion. The government’s signature Tax Watch campaign uniquely went beyond publishing raw data to placing “weekly full-page ads in domestic newspapers [in print and online] that combine killer statistics, intuitive infographics, and compelling visuals” (p.60).

Producers also need better intelligence on which channels are most effective to reach different user groups. Masaki et al. (2016), in their study of governance data use, find that a one-size fits all strategy may not be possible as “the channels that leaders use to find governance data vary by where leaders work” (p.10). AidData and its consortium partners in the AidData Center for Development Policy are shedding further light on the efficacy of different outreach modalities by testing the responses of prospective users of the aid information management system in Timor-Leste to a 3-minute video spot introducing the system’s functionalities and an email newsletter with analysis and visuals from the system.155

Section 6.1.2.3
Principle #6: Training will be critical to overcoming the capacity gap, but funders and producers must be more systematic in testing what does and does not work

Even when government, development partner, and civil society organization representatives are aware of the data available to them, they may still lack the capacity to confidently access and use this information. As Paris21 (2016) argues, equipping staff in national statistics offices and other domestic institutions with the skills to “identify, evaluate, and access new data sources” will be critical to realizing the data revolution for sustainable development. They also offer some promising ideas for how to serve multiple audiences simultaneously, such as: setting up centers of excellence to pool training resources, publishing Big Data sandboxes for people to explore relevant data sources and tutorials, and leveraging partnerships with academia and the private sector to support curricula design and deliver training (Paris21, 2016).

Training is also an underutilized channel for individual producers to increase awareness of their data products and solicit feedback from prospective users. Interviewees across the three countries frequently expressed interest in receiving additional training in the use and analysis of data to support their work. If generic launch events and media coverage allow producers to broadly advertise their data to a general audience, in theory, customized skills trainings and focused consultations are an opportunity to go deep with key user groups.

While prospective users have a clear interest in more training opportunities, not all training may be equally effective in promoting long-term data use. There are many critical decisions that go into the design of a one-off training event or a longer-term capacity-building program – who participates, how is training delivered, what mechanisms are in place to ensure follow-up, to name a few. Surprisingly little is known about the attributes of training and capacity-building efforts that are most likely to succeed.156

AidData and its consortium partners in the AidData Center for Development Policy are helping to close this evidence gap through conducting field experiments in Timor-Leste and Honduras to assess whether and how participation in a training event changes usage patterns of the aid information management system among domestic policymakers and practitioners.157 Funders, data producers, and researchers should build upon this evidence base and conduct additional experiments to test how users respond to different formulations of data training and statistical capacity programs to improve the returns on their investments.

Section 6.1.3.1
Principle #7: Producers of data solutions should seek to enhance, not replace informal knowledge sharing via social networks

Government, development partner, and civil society organization (CSO) leaders have a natural tendency to source information from those they know due to the confluence of habit, preference, and lack of perceived alternatives. Producers of data solutions should attempt to support these robust human information networks, not replace or compete with them. Specifically, they could proactively enlist the support of a sector working group or an association of CSOs to adopt centralized (and open) data systems in a mutually beneficial alliance.

Producers could collaborate with network members to ensure that their databases are capturing up-to-date development cooperation activities for all actors. As network members become more involved in curating and adding to central data repositories, this is more likely to increase the confidence of prospective users in the comprehensiveness and utility of government data. In addition, producers could seek out the assistance of sector working groups or well-respected professional associations as third-party validators to publicly certify or give their seal of approval for the data’s accuracy.

Sector working groups or professional associations could partner with official data producers for help in documenting qualitative lessons learned for future use. Official data producers could systematically integrate this information as a value addition in centralized repositories that informal working groups or associations do not have to maintain. As official producers grow in their data management capacity, they could provide critical assistance to networks with new tools, methods, and approaches to organizing qualitative insights that are often unstructured, subjective, and context-specific.

Section 6.1.2.3
Principle #8: Funders and producers should work with the grain of human behavior to align incentives in favor of data-driven decision-making

For many government, development partner, and CSO leaders, analysis of data and its use in the context of decision-making are viewed as “overtime activities” (Development Gateway, 2016, p.3). This response is unfortunate, but understandable, as policymakers and practitioners are guided by an internal cost-benefit calculus. The informal norms and formal rules governing decision-making in government agencies or organizations seldom reward those who make data-driven decisions or penalize those that do not. Therefore, the perceived costs in time, effort, or lost political capital to using data are known and proximate, while the perceived benefits of making decisions based upon evidence are uncertain and distant.

Regardless of whether one is a skeptic of, or advocate for, the post-2015 development agenda, it is clear that this process offers a compelling window of opportunity to change the “rules of the game” (North, 1990, p.3). If funders and producers of data and statistics want to create the conditions for evidence-informed policymaking, they need to find a way to “work with the grain” of human behavior (Levy, 2014, p.7).158 In other words, they must crowd in, rather than short-circuit, the interest of political actors in favor of using data as they allocate resources, target projects, and evaluate the performance of development programs (Custer et al., 2016).

Monetary and non-monetary rewards should be employed to increase the perceived benefits for government, development partner, and civil society leaders to use development data. Executive branch-led transparency initiatives in countries such as the Philippines have tied performance bonuses and merit increases to adherence with specified disclosure practices (Custer et al., 2016). Funders and producers of development data could work with senior leaders in forward-leaning government agencies and organizations to develop a corollary to tie performance bonuses to behaviors that exhibit effective use of data to inform decision-making. In addition, senior leaders could explicitly reinforce cases of positive deviance within their agencies and organizations by prioritizing access to training, recognition, and advancement opportunities for decision-makers who use development data in their work (Zeitlin et al., 1990; Development Gateway, 2016).159

Section 6.1.3.3
Principle #9: Producers should be more transparent about their data collection methods and quality assurance procedures to get a credibility boost

The determinations that policymakers and practitioners make, regarding whether a given data source is credible, are critical to whether they ultimately decide to use this information in their work (Masaki et al., 2016). Unfortunately, prospective users perceive many data sources, particularly those produced by governments in low- and middle-income countries, to be untrustworthy and unreliable. As previously discussed, ensuring the accuracy of development data is indeed a problem. However, there is likely more behind this trust deficit than a lack of accuracy alone. Distrust of official data sources could also reflect low public confidence in the government overall.

Opaque methods for collection and validation do little to alleviate concerns that producers are manipulating data behind the scenes for their own political ends. Increasing public trust in development data will require not only technical fixes to improve the accuracy of the data, but also a political commitment to greater openness and transparency about what is being measured, how data is collected, and what procedures protect this information from tampering. If credibility is indeed a major barrier to use, data producers, both official and unofficial, have everything to gain from becoming more transparent in documenting their assumptions, methods, and processes for managing data.

Funders of data initiatives and senior leaders in governments and organizations should reward and reinforce the efforts of producers to become more transparent in their methods and assumptions. In doing so, they will help create the conditions for more robust data use. Public perceptions may lag behind policy changes, but transparent producers are more likely than their stubbornly opaque counterparts to win back the confidence of prospective data users.