Data Debt: How Data Hoarding Is Silently Choking Your Organization

Have you watched the A&E TV series Hoarders? It’s currently in its tenth season. I’ve watched a couple of episodes, and I couldn’t help but draw a parallel between how these hoarders keep useless junk in their homes and how organizations do the same thing with data. Organizations amass vast amounts of data in their infrastructures — some useful, some not. In the end, both hoarders and corporations are frustrated and defeated and eventually seek help to manage the madness.

Unlike the people featured in Hoarders (and have a compulsive hoarding disorder that can be helped through therapy), organizational hoarding of data is a dangerous combination of the lack of IT emphasis on data, the lack of vision for data usage and the lack of funds and resources to clean it up. Most importantly, this useless data creates what we call “data debt.”

What is data debt?
Data debt is the result of not having a detailed, written plan for data, including its collection, maintenance, storage, and disposal. On January 1, 2020, a new law, the California Consumer Protection Act (CCPA), put a new emphasis on data management. These new laws mean that data collected in the early 2000s may no longer be usable because of the privacy promise made to customers when the information was collected. Data collected today will have different requirements for usage. These new laws, like the General Data Protection Regulation in Europe, also put limits on the amount of time that data can be kept and provide the consumer with the right to be forgotten, meaning that a contact request to be dropped from the database must be honored. Storing and archiving data in most organizations is not a high priority. Because memory is inexpensive today, organizations often end up keeping data for longer than they should. Storing useless, unused and incomplete data compounds the data management issue. Managing this data debt is made worse when organizations convert from one financial system to another (for example, from Oracle to SAP), or when organizations change business models, such as moving from on-premise to SAS. Without a plan for dealing with data, data debt piles up.

Because data is a perishable asset, data debt becomes increasingly harder to manage as it ages. It is estimated that any data set will erode at 25% a month. With that fact in mind, how much data in your organization is too old to be useful?

What does data debt look like?
If we have the funds and we think the data we are keeping may be useful someday, why do we think data hoarding is an issue? What problems does it actually cause? And is data debt to blame for our “data” problems?

Check these situations where data debt might be involved:

• Slow response times that take weeks, instead of minutes, to create and generate reports. Depending on how datasets are structured, reports are often having to look through massive amounts of data to derive information. Either you invest in more powerful machines to manage data faster, or you organize your data so that it can be easily selected. Both options are investments of money and time.
• Inability to get the right data at the right time to the right person who could use it.
• Extended schedules, due to data cleanup, which happens most often when companies are converting from one application to another.
• Inability to use data effectively to draw insights, create accurate reports, or develop workable artificial intelligence or machine learning applications.
• Loss of organizational productivity.
• Inability to complete a digital transformation.

Data debt erodes trust
Data debt will eventually lead to several problems for organizations. Like technology debt, (which refers to the cost that an organization accrues because it has either deferred technology decisions or chosen a cheaper or quicker solution to achieve something else), data debt can cost the organization big time in the long run.

For example, consider a company that wants to move its aging on-premise accounts receivable system to a more efficient cloud-based system. It’s been estimated that the number one cause for failure in system conversions is the quality of data transferred between applications. This loss can cost the organization time and money when schedules are missed, or worse still slipped altogether.

All these issues with data debt eventually lead to frustration from the ranks of senior management because they aren't getting value from their data. Once frustration sets in, executives, management, and users will begin to lose confidence or trust in the data. And lack of trust within an organization slows productivity and growth.

But all is not lost. There are ways to bring trust back to your data similar to the Hoarders series when specialists arrive to help understand why the hoarding is happening and then help restore the home to a maneagable state. Organizations must first assess whether they have data debt, evaluate the data debt identified, plan the data cleanup program, set a goal and measure progress toward data debt elimination or consolidation and finally develop a program to keep the data clean moving forward.

Learn how NTT DATA can optimize your business with data insights and AI.