Data should be a first-class citizen in the cloud

“It’s all about the data!” has been the battle cry for traditional systems architects as they fumble through building complex systems—specifically, how they will generate and process data as a core capability.

It’s all about the data, from customer behavior analysis to predictive modeling. However, the cloud ecosystems, including the push into the clouds, often take this asset for granted. Should they? How does data affect the usefulness of these systems? 

Let’s cover a few common problems and how you can improve the business value of your data. 

Inadequate data governance

Data governance is often overlooked and needs to be understood within enterprise IT. Please don’t take my word for it; ask somebody what data governance systems are in place and stand back to watch the confused looks. 

Cloud platforms provide robust infrastructure and services but often need comprehensive mechanisms for data management, privacy, and security. The solution is as easy to understand as the problem. Organizations must take responsibility for implementing proper data governance frameworks, both policies and systems. Also, cloud providers must prioritize and streamline these capabilities to ensure data protection, compliance, and ethical handling.

Lack of interoperability

Data is often tied to specific cloud platforms or services or is coupled. This makes migrating or integrating with solutions difficult. Indeed, this is the foundation of most data problems: the creation of data silos that are difficult to break down.

Data should be treated independently of the underlying cloud infrastructure. This enables movement and integration across various platforms. I’m not sure why this message hasn’t been heard; I’ve been teaching the techniques and mechanisms of how to do this for years, certainly here. However, it’s the most violated rule that I see, and as we move into cloud-based platforms, it’s going to become a much bigger problem.

The only way to solve data interoperability problems is to design the solution into the systems. Interoperability is typically not something that can be fixed with “bolted-on” technology, which does little more than make things more complex.

Insufficient data access and control

A close cousin of the interoperability problem, data access and control are limited in many cloud environments if not designed properly and can prevent organizations from truly harnessing their business data. There doesn’t seem to be a middle ground here; either data is entirely accessible or not at all. Mostly, the controller is turned off and valuable data goes unleveraged and systems are underoptimized.

You only need to look at the rise of generative AI systems to understand how this limitation affects the value of these systems. If the data is not accessible, then the knowledge engines can’t be appropriately trained. You’ll have dumb AI.

This lack of control is due to opaque data ownership models and limited data processing and storage control. The solution is for organizations to create greater transparency and control over their data. This includes defining access privileges, managing encryption, and deciding how and where data is stored. This would ensure that data owners retain sovereignty and information is still available.

Data is not a first-class citizen in the cloud systems that are being built right now, and it should be. The tools for data governance, interoperability, and access are well known, and the processes to leverage them properly are well understood. For some reason, enterprises don’t want to bother.

You can certainly push past data issues and hope nobody notices, but the reality is you’re extracting only a fraction of the value that you could get from the same systems. As AI comes into play for most enterprises, the value of data is no longer just a concept, it’s a reality you can’t avoid.