Exploring the sixth data platform: How AI is shaping the future of enterprise data management – SiliconANGLE News

In a year marked by artificial intelligence changing everything, the journey toward intelligent data applications has emerged as a key storyline. That journey includes the emergence of the sixth data platform — something that could redefine data utility and application intelligence.

“It’s a complete generational shift. This is not your father’s internet anymore,” said theCUBE Research industry analyst John Furrier on a December edition of theCUBE Pod. “That’s the rundown; AI everywhere, anywhere, all the time. That’s what’s happening.”

Over the past year, theCUBE has been tracking the story of how AI is shaping the future of enterprise data management. It’s why when gen AI was still relatively fresh in the news, Greylock partner Jerry Chen said in a July LinkedIn post it was “AI or die” for most businesses. At the same time, the data showed AI was taking spending priorities away from other areas, according to theCUBE Research chief analyst Dave Vellante.

“It’s definitely impacting other sectors. Even cyber, cloud is being optimized. RPA has come down a little bit, even though it’s now ticking up again. Automation’s doing a little better,” Vellante said in a July edition of theCUBE Pod. “I think AI is stealing not only mind share, but also wallet share and resources away from these other projects. People are delaying some of their other IT projects, focusing resources on AI. That’s what the data says.”

Since last summer, AI’s gradual takeover has continued to play out — and projections suggest that when it comes to enterprise data management, big changes are on the way. Gartner Inc. has forecasted that more than 80% of enterprises will have used generative AI APIs or deployed generative AI-enabled applications by 2026, up from less than 5% in 2023.

“Generative AI has become a top priority for the C-suite and has sparked tremendous innovation in new tools beyond foundation models,” said Arun Chandrasekaran, distinguished VP analyst at Gartner. “Demand is increasing for generative AI in many industries, such as healthcare, life sciences, legal, financial services and the public sector.”

So, how is AI transforming DevOps and enterprise data management strategies? And what are the implications of the emerging sixth data platform and intelligent data applications?

Explaining the sixth data platform

Emerging AI-infused applications require real-time processing of massive, diverse data sets. But the current modern data stack — built on the principle of separating compute from storage — is being challenged by these use cases. That’s why a change in underlying data and data center architectures is necessary, especially regarding exabyte-scale workloads, according to Vellante. It means the principle of separating compute from storage must move to separate compute from data.

“And, further, enable compute to operate on a unified view of coherent and composable data elements,” Vellante wrote in a recent edition of his Breaking Analysis series. “Moreover, our opinion is that AI will be used to enrich metadata to turn strings, such as ASCII code, files, objects and the like, into things that represent real-world capabilities of a business.”

Vellante has been exploring the semantic layer in-depth since early in 2023. In a January 2024 edition of his Breaking Analysis series, he wrote that the “future of intelligent data apps will enable virtually all organizations to operate a platform that orchestrates an ecosystem similar to that of Amazon.com Inc.”

That means dynamically connecting and digitally representing an enterprise’s operations, including its customers, partners, suppliers and even competitors. It includes the ability to rationalize top-down plans with bottom-up activities across the many dimensions of a business, including demand, product availability, production capacity, geographies and more, according to Vellante.

“Unlike today’s data platforms, which generally are based on historical systems of truth, we envision a prescriptive model of a business’s operations enabled by an emerging layer that unifies the intelligence trapped within today’s application silos,” he said.

Exploring the shift to intelligent data applications

Starting in late November, Shelly Kramer, theCUBE Research managing director and principal analyst, began a video series titled “The Road to Intelligent Data Apps.” It was an exciting launch given that theCUBE sees the road to intelligent data apps as truly being the next frontier, according to Kramer.

The last decade in data analytics was all about cloud architecture and the process of separating compute from storage. The modern data stack lakehouse became the historical system of truth, and the industry saw the rise of simple, scalable cloud platforms, such as Snowflake and Databricks.

“The next decade, though, we anticipate some massive change,” Kramer said. “This decade will be all about separating compute from data. We’re evolving to what will be a new historical system of truth, with more easily composable data products that will evolve into applications. It’s something we believe will be an even more profound shift than the MDS lakehouse.”

While what that exactly looks like remains unknown, what is known is that the combination of data and metadata will form this new system of truth, according to Kramer. When it comes to enterprise software, this year will likely be the one where key milestones are met in a once-in-a-decade shift in data architectures, according to theCUBE industry analyst George Gilbert.

“This is probably the year where we start to see this emphasis shifting from what was the modern data stack, really that Snowflake defined, that supported analytics by separating store from compute and making it cloud-native so you could have as much data as you wanted and independently scale the compute,” Gilbert said. “But we need something different now. This is where we’re going to see this equally profound shift that’s going to ultimately support intelligent data apps.”

Overcoming data platform incompatibilities

This all poses a question: What is actually involved in building the modern data platform? Customers have always used multiple vendors, and it’s important that they’re able to continue to do that, according to Bob Muglia, the former CEO of Snowflake Inc.

“The platforms are maturing; each one of them is different. They all do very similar things. They’re trying to provide very similar services to customers, but they are doing it in somewhat different ways,” Muglia said on an episode of The Road to Intelligent Data Apps. “I think each one of them will continue to improve what they can provide in terms of governance and every other aspect of their platform.”

They are also accommodating independent software vendors building on top of it. Essentially, these superclouds or data clouds being put together are platforms on their own, Muglia added.

“There’s clearly another layer that’s been built on top and that’s been targeted by ISVs, and that can be selected by customers,” he said. “There’s some real major impediments to cross-vendors solutions. The one that I’m most focused on at the moment is that what’s clear is that the data lake is emerging as the place where data is stored, by and large, data of all kinds.”

That includes structured and semi-structured data, but it also includes videos and documents and everything else, including text data that is of great interest to people as they begin working with large language models. What is emerging now is a set of technologies that essentially unify and provide a coherent way of looking at data from both a file perspective, as well as from a table perspective, according to Muglia.

“Unfortunately, there are three different standards that exist to this today, and customers have to select what they’re working with, and there are some significant incompatibilities between them,” he said.

It’s a bit like the Betamax versus VHS situation, in terms of the fact that customers are putting their data into a given type of data lake. While the data storage in all of these is the same, they’re all essentially using Parquet files, according to Muglia.

“Fortunately, the data itself is compatible. The metadata associated with the data, the metadata that turns these files into tables is all different in its structure,” he said. “Now, the unfortunate thing about that is, it means that these systems don’t work together the way they should. The positive aspect of it is the metadata is quite small relative to the size of the data. It may be possible to just translate and transform it across that. And we’re beginning to see some solutions.”

The future of enterprise data management is slowly taking shape

There are still countless open questions about how the intersection of generative AI, cloud computing and data transformation will reshape industries. But as theCUBE has hosted events such as Supercloud 4, which examined how these advances are impacting every industry, it became clear to Furrier that GenAI represents the biggest wave we’ve ever seen.

“The idea of the super-structures of data, the super apps that are coming out with AI … it’s going to scale labor and reduce the costs and increase the creative intellect,” Furrier said as part of a show wrap for Supercloud 4. “AI scales intellect and scales data. This is a generational movement. It’s a revolution, in my opinion.”

All told, the revolution won’t happen overnight. But the bubbling up of trends — including the sixth data platform, data mesh, data fabric and the end-to-end intelligent enterprise — all underscore what is changing customer needs, according to Vellante.

“In this future vision, elements of a business are represented digitally and in near-real time. It won’t happen tomorrow and will evolve over a decade or more,” he wrote in a January edition of his Breaking Analysis series.

Like flying a plane on instrument flight rules, customers will need to gain confidence in these systems. They’ll need to rationalize nonintuitive recommendations and trust that governance and privacy are integral to the system, according to Vellante.

“Regardless of the challenges, the business value impact of unifying intelligence across disparate systems will be enormous,” he said.

Image: This image was generated with the assistance of AI