Dizzying Data: What the Rise in Data Complexity Means for Business

The rise in data complexity

The first thing that needs to be acknowledged is that today’s data is much more complex than the data a typical business was dealing with 20 years ago.

Data complexity ─ or the measure of how complex data is ─ describes large data sets from disparate sources and needs many resources to process. Often, complex data comes from several sources, with each having a different structure, size, query language and type. Of the other different types of data (simple, big and diversified), complex data can be described as a combination of big data and diversified data.

The technological advancements that allow businesses to gather, store and access vast amounts of data result in the availability of complex data, which is collected both internally, during the business’s regular operations and externally.

Alongside the size of the data, today’s data is often diverse and is no longer confined to spreadsheets. Various automated systems generate large amounts of structured or semi-structured data, such as in the case of machine data, social network data or data generated by the Internet of Things.

Today’s businesses need to analyze more data ─ and more data sources ─ than ever with the exponential growth of new platforms, online channels, software and other technological innovations.

The fact that data has grown both larger and more diverse creates new challenges in the world of data analytics. Today’s datasets might each follow their own particular logic and require a deeper understanding of the way data is extracted and structured before any serious analysis can be done (again, when compared to the traditional spreadsheet).

To illustrate the breadth and versatility of complex data, consider a pharmaceutical company. If a department within the company wants to analyze every internal research activity about diabetes, there’s seemingly limitless information to compile ─ from genetics to insulin levels to patient demographics. Data complexity analysis enables the researcher to organize the datasets into a structured view of related concepts to visualize how they connect.

While this challenge exists across the organization, it could be particularly vexing for business professionals, who typically do not possess a technical background.

Data-driven business practices

While data is becoming more complex to understand, organizations are demanding broader use of it in an increasing amount of business scenarios. Once the data is available, senior management looks to put it to use by promoting a more data-driven corporate culture.

However, this intensifies the difficulty for employees: Not only are they required to deal with much more complex data, but this requirement often becomes a critical part of the way their performance is reviewed. In this state of affairs, the nontechnical manager could feel as though he has not been given the proper tools to succeed, which is why data access is so important.

Here are several ways you can leverage data in your business.

Data-driven decisions

Managers are often asked to present cold, hard data points to support their proposed decisions or to justify previous ones. In addition, a majority of executives in leading enterprises report that analytics is central to their overall business strategy. The approach makes sense as there’s not much point in collecting copious amounts of data if you’re not going to use it when your organization reaches a crossroads. Data allows businesses to make highly informed decisions for the best possible outcome ─ whether that result is a better bottom line or creating a stronger company culture.

Results-based reporting

Feel-good, fluffy pie charts or upward-trending graphs littered throughout a presentation will no longer suffice: Today’s executives must truly show that their actions have had a positive, quantifiable effect. Evaluating data remains one of the top ways to convince teammates, department heads or even yourself to take action.

Data access

Granting access to data spans business departments and disciplines. Usually, the operations and information technology (IT) departments have the most technical information, but investing in becoming a data-driven business means departmental collaboration. This may mean enabling access to new software, sharing reports or holding meetings to discuss concrete numbers.

Data should be accessible to the people who need it to perform their tasks and responsibilities. Often, marketing shares its data with other departments to illustrate business decisions. Research and development will share data with marketing or production to demonstrate what is realistic and feasible versus impossible to accomplish.

Each employee and department needs to understand the importance of data to change the culture from data-phobic to data-driven.

Data-positive culture

Cultivating a data-positive culture requires choosing the right company software, investing in security and leading from the top.

In choosing the right software, considerations for the technology include the following:

• Is it scalable with the business’s intended growth?
• Is there a self-service option?
• What type of data/analytics reporting does it offer?
• How can I connect my other datasets to the data this technology will gather?
• Does it support collaboration?
• Is it difficult to learn?

Accessibility makes all the difference in choosing the right software. The learning curve, especially in pulling data and insights, should be gradual enough for every intended user to understand.

To establish a data-positive culture, ensure customers and employees understand how the data is encrypted and kept safe. Executives should model the use of this technology and data complexity to encourage the rest of the company to use it too.

Ongoing education

As with most technology, data analytics continues to evolve. Establishing data-first business practices means a commitment to continuous education and improvement. Employees will learn how to use data in their roles, managers will learn how best to leverage data for important business decisions and teams will learn how to communicate their data needs to each other through collaboration. [Related article: Should you make business decisions on gut instinct or hard data?] 

Businesses that provide education and training on using data will benefit from companywide knowledge and a shared value of data.

How to use complex data in your business

Complex data may seem like an insurmountable challenge in business. Should executives give up on data altogether and leave it to IT professionals and business analysts or should every manager be trained to be a “mini-business analyst” capable of understanding and producing insights from increasingly complex datasets?

The answer lies somewhere in the middle. Managers must become more data-savvy and learn to understand basic concepts in data analysis and visualization, in effect, to be able to tell a story with data and to dig into the relevant information and reach data-driven insights.

Tips for using complex data in business

• Define what you want to measure with data clearly: One of the biggest pitfalls of using complex data is a goal getting marred by the sheer volume of aggregated data. Before launching a campaign or pulling numbers, sit down and define what the end goal of data measurement is. Ensure everyone on the team understands and keeps track of relevant key performance indicators.
• Understand how to use unique IDs: With the amount of complex data you’ll be working with, it’s important to understand how unique identifiers denote customers. This unique ID will match a customer or client across multiple software platforms, so knowing how to assign the ID, how to avoid duplicates and how to pass the ID structure through to every system in use ensures clean, helpful data.
• Organize data in a simple structure: Try to organize data in the most efficient, lean structure to avoid headaches down the road. Cover what you need for financial reporting and beyond that, keep to a simple structure that will make working with datasets easier.
• Establish a central source of truth: Along the lines of understanding unique IDs, having a source of truth enables you to answer questions as they come up in daily business issues. Where will you look if a client misses a payment or claims they were double charged: your accounting software, a manually created spreadsheet or a customer relationship management system? Which accounting software you choose doesn’t matter, as long as you and your team are aware of the single source of truth in the business.

Across the board, avoid muddying data with inconsistencies, working off of multiple sources of truth and changing unique customer identifiers.

Today’s self-service data analytics tools certainly go a long way toward enabling the use of complex data by simplifying the process of combining multiple sources of data to reach new and unexpected conclusions as well as presenting the results in attractive visual formats.

Both the line-of-business workers and their companies should take it upon themselves to move their organizations past spreadsheets and into the new generation of business intelligence software while making sure that the selected tools will be able to meet the demand for rapid analysis of complex data, even by nontechnical users.

Tasks such as managing complicated data schemas, setting up automated extract, transform, and load processes and ensuring data quality and governance are difficult tasks that are best left to professionals. The same applies to applying advanced statistical and predictive models for deeper analysis.

It would not be beneficial to train business users to perform these tasks. Instead, companies that want to get serious about data should realize that today’s big and disparate datasets require dedicated employees and the tools to help them prepare and analyze complex data.

Solution to data complexity

The solution to the problem of growing data complexity in today’s data-driven enterprise is threefold:

1. Educating business departments to become more familiar with data and analytics.
2. Implementing tools that enable business users to perform their own analysis.
3. Tasking business analysts with the heavy lifting of data modeling and advanced analysis.

Once the business side of a company is more familiar with the data, it will also be able to communicate much more freely and easily with the technical side while understanding what can and cannot reasonably be done with the organization’s existing data.

This will promote harmony further within the organization and pave the road to a brighter, more data-driven future.

Kimberlee Leonard and Saar Bitner contributed to this article.