Using Inspektor Gadget for Kubernetes observability

Platform engineering is becoming a compelling concept for enterprises, as they’re devoting increasingly large amounts of resources into cloud-native application development. It doesn’t matter if you’re using your own Kubernetes instances in your data centers, or working with managed environments in public clouds, you’re going to need to understand how they’re operating, taking advantage of a new generation of observability and security tools.

Container orchestration tools like Kubernetes are hard to manage using traditional tools. As they instantiate new containers based on resource usage or in response to events, their behavior is not predictable. If you’re using an agent-based monitoring tool, then agents need to register with monitoring tools as containers start up, and de-register as containers stop. The process adds complexity to tools, and it’s hard to use the resulting data to get a complete root cause analysis, or to trace an anomalous activity.

There are cloud-native concepts that help, like observability, but it’s becoming clear that we need a way to step outside the orchestration environment, while still tracking its compute and networking operations. With containers hosted by modern hypervisors or by low-level container daemons, we need a  lower-level way of monitoring systems.

Seeing from inside the Linux kernel

In the past that would have involved working with kernel-level code, adding your own or third-party kernel modules, and recompiling the Linux kernel each time a new tool or an update comes along. If you’ve ever compiled Linux from scratch, or written a kernel module, you know that it’s not the easiest of tasks—and often it’s extremely time-consuming. If you’re using a ready-to-run image from a cloud provider, you’re unlikely to have the necessary permissions or access.

Luckily, we have an option that avoids all that complexity, giving us controlled access to kernel operations, in the shape of eBPF, extended Berkeley Packet Filters. With eBPF we can inject code into the Linux (and Windows) kernel, using a verifier to ensure security. Hooks in the kernel generate events that trigger eBPF programs, allowing access to networking and other low-level operations.

This approach allows monitoring tools to get data on the operations of the hosted Kubernetes environment, tracing key operations and allowing external applications like Prometheus, Cilium, and Retina to log the data, tying it to individual containers and to the applications that are orchestrated by Kubernetes. With eBPF support in the Linux kernel, you can use it to monitor not only physical instances on your own hardware, but also virtual machine hosts on a cloud service.

Introducing the Inspektor

Microsoft subsidiary Kinvolk is best known for its Flatcar Linux distribution, but it’s rapidly becoming one of Microsoft’s open-source centers of excellence. Whereas Deis Labs focused on cloud-native development, Kinvolk is more aligned with platform engineering and operations, providing tools like the Headlamp Kubernetes UI. Over the last five years or so, Kinvolk has been working on a set of eBPF monitoring and capture tools, bundled as a suite of kernel “gadgets” with user space management and display services.

Named after the part-robot French detective, the Inspektor Gadget project offers a growing set of gadgets, as well as a framework for developing your own. The mix of gadgets ranges from tools that audit your Kubernetes configurations to others that capture performance information about your cluster, either for profiling or for real-time analysis. The most useful set allows you to trace a number of key metrics, including low-level networking operations. Your issue might not be DNS, but Inspektor Gadget can help you prove that it isn’t.

While tools like Falco and Retina are intended for longer-term observations of a Kubernetes infrastructure, a stand-alone install of Inspektor Gadget is most useful for ad hoc investigations and explorations. That’s not a bad thing. Platform engineers need tools that can be used quickly to diagnose specific problems, without having to invest time in setting up and configuring a complete observability solution.

With Inspektor Gadget you can get the information needed to show what’s going wrong and why, and then explore potential causes, all from the Kubernetes command line. The data you get back, while low-level, is associated with namespaces in your Kubernetes cluster, allowing you to quickly isolate specific pods and containers.

You can use Inspektor Gadget on its own, or through a tool like Microsoft Defender for Containers, which recently added a sensor component that uses Inspektor Gadget tools to look for threats like container escapes. These are some of the riskiest compromises for Kubernetes, as they allow malicious code to break container isolation and access the host OS. Microsoft is using Inspektor Gadget to replace Defender for Containers tools that were previously powered by Sysdig’s eBPF probes.

Running Inspektor Gadget in your cluster

Getting started with Inspektor Gadget is easy enough. The toolkit installs both a kubectl plugin and a daemonset in your Kubernetes cluster. You can use the krew installer tool to install the kubectl tools, or install it manually by downloading a binary or compiling the source code and then adding the resulting executable to your path.

Once you have the gadget command-line tool installed, you can deploy the daemonset using its deploy command, creating a pod for your gadgets and applying the appropriate security controls. You can deploy the daemonset to your entire cluster or to a specific node or nodes. It can then be configured to start alongside new containers. To uninstall, simply use the undeploy command.

You don’t need to install Inspektor Gadget in Kubernetes. It can be installed as a Linux application on a host server, allowing you to trace containers from outside Kubernetes. While this approach isn’t suitable for a cloud-hosted Kubernetes, it can be useful for monitoring experimental installs and on-premises clusters.

Once installed you can use Inspektor Gadget’s gadgets to install eBPF code in your host OS kernel, attaching the gadgets to specific kernel functions. The eBPF code runs when specific syscalls are made, allowing the gadget to trace the operation.

Trace results are written to kernel buffers, and then read back by the userspace gadget code, displaying streamed data through the Kubernetes CLI. In the background a tracer manager keeps track of the various gadgets in use, with a map of the containers and namespaces they’re associated with, so data is associated directly with a specific container. Results are filtered by container and by event, giving you a way to quickly dive into the events you want to monitor.

Inspektor Gadget provides a Prometheus gadget that delivers metrics to a Prometheus instance, allowing them to be visualized with tools like Grafana or analyzed by other Kubernetes management tools. Support for Prometheus allows you to run gadgets in the background, especially if you’re using network tracers or profiling tools to get a picture of the overall performance of an application and a cluster, as well as the underlying network.

Extending the gadgets

Usefully Inspektor Gadget can work with other eBPF programs, adding additional information to basic queries. You can write a simple eBPF probe to look for a specific event, and then call Inspektor Gadget to add details of the current mount namespace to the event log. This way you can quickly add code to look for new critical vulnerabilities, allowing you to pass that data to your monitoring application via the Inspektor Gadget CLI.

Tools like Inspektor Gadget are important for dealing with the unpredictable nature of cloud-native applications. By using eBPF we can get down into the weeds of complex networking meshes and understand exactly what is happening when containers communicate with each other, allowing us to build the right infrastructure for our code. There’s a lot to like here, both in how Inspektor Gadget integrates with familiar Kubernetes tools and in how it can be extended with your own gadgets to address your own issues.

As Kubernetes matures, it’s important to have ways of seeing what happens under the hood. Without that data we can’t be sure that we’re delivering the enterprise-grade architectures that are essential to support our code and our users. eBPF and Inspektor Gadget are an important way of delivering that maturity, alongside observability tools that allow us to interpret and analyze the data and deliver a manageable cloud-native platform.

Also by Simon Bisson:

• Microsoft Azure eases Kubernetes development with a new Draft | Jun 08, 2022
• A peek into Azure’s Kubernetes container pattern future | Aug 07, 2018
• Kubernetes at the edge with Akri | Oct 27, 2020
• How to use the Kubernetes C# client library | Mar 17, 2020
• Securing the Kubernetes software supply chain | Dec 15, 2021