Some observations on cloud computing observability

The term and the concept of observability aren’t new at all. Observability first appeared in the world of engineering and control theory. Although the definition depends on whom you’re asking, some common patterns and understandings are emerging. Mostly it’s a new buzzword by the AIops providers and some thought leaders in the cloudops space (meaning me). 

At its essence, observability is the measure of how well internal system states can be inferred from knowledge of all external data and states. It’s a bit different than monitoring which is something you do (a verb); observability is an attribute of a system (a noun).

If that’s a bit confusing, here’s a more pragmatic definition: the ability to leverage the concept of observability and the tools that support that concept. Most monitoring and operations tools (such as AIops tools) claim some role in observability. However in some respects, to me it’s the same old bourbon (monitoring) in a much fancier bottle (observability). I look at it with a bit of well-placed skepticism.

On the bright side, more tools are being purpose-built for the notion of observability, including most of the emerging AIops and cloudops toolsets, as well as other monitoring platforms that deal with complex operational data, such as security, database, and network monitoring. The purpose of observability is leveraging all of these data points holistically, which most enterprises don’t do these days.         

Observability is important to managing and monitoring modern systems, considering we deal with both modern applications and the pace at which these applications are delivered. The widely held practice of bolting on monitoring and management after applications are deployed does not scale very well.   

Part of the notion of observability is the need for up-to-date instrumentation to better understand the properties, performance, and behaviors of an application. We need tools to deal with underlying complexity and distribution. Of all the barriers to the cloud moving forward, what scares me most is that we’ll build too much complexity with no clear way to manage it at scale.    

Basically, monitoring has these attributes:

• Passive consumption of information from systems under management to determine state and status.
• Dashboards provide information about the state of systems and rely on humans to respond.
• Tools are purpose-built for static systems and platforms that don’t change much over time.

Observability takes a different approach: 

• Holistically considers all data from systems under management to determine a proactive and automated response.
• Asks questions based on a continuous understanding or hypotheses, continuously improving how systems are observed, tuned, and fixed using automation.
• Purpose-built for dynamic systems and platforms with widely changing complexity. The tool adapts to the systems under management, not the other way around.

To be honest I take these new concepts as something to consider, not to follow religiously. The history of IT is full of ideas that seemed important for a time and faded away, never to be heard from again. However, considering the importance of operations and tools than can allow us to operate under increasingly complex environments, this concept could go wide.