Securing Azure Kubernetes with Falco

Falco, the open-source, cloud-native, runtime security tool, recently graduated from the Cloud Native Computing Foundation’s incubation program. That means it’s considered stable and ready for use in production environments, including Azure. It joins many of the key components of a cloud-native platform including Helm, Envoy, etcd, KEDA, and Cloud Events.

I recently had a conversation with Loris Degioanni, the CTO and founder of cloud-native security company Sysdig and the creator of Falco, about the philosophy behind the project and how it’s being used across Kubernetes applications.

Why Falco?

There’s a need for security tools designed to work in Kubernetes and in containers. Microservice environments that scale up and down on demand are a long way from the monolithic applications of old, or from more recent architectures, like n-tier or service orientation. New instances need to be detected and monitored as soon as they spin up, ensuring that as our service meshes become larger and more complex, we have the visibility we need to keep them secure.

You can think of cloud-native security as akin to the role of observability in devops. We’re not looking for specific incidents, but for patterns in telemetry data that show exceptions to the norm. Traditional security tools might help protect code running in a container, but they’re not able to handle serverless patterns or dynamic infrastructure.

Sponsored by cloud security company Sysdig, Falco is a runtime security tool that’s designed to operate at a very low level in your containers and applications, with direct access to kernel-level networking functions. The result is the ability to detect intrusions and compromises in real time across your entire application infrastructure. Event data is collected and a rules engine is used to identify security issues. You can build your own rules, or work with community-developed rules.

Falco’s rules are best thought of as policies. As Degioanni says, “[Falco has] a policy language that you can use to define a policy, like somebody starting an interactive shell in one of my Redis containers.” When a policy is matched, an event is generated and delivered to a central monitoring service and used to deliver the appropriate response.

One key tool for Falco is its use of eBPF probes. These are sandboxed scripts that run inside the Linux kernel, providing direct monitoring of syscalls at speed. This should help identify intrusions quickly, allowing you to automate responses using external applications, for example throwing a cloud event and triggering serverless actions that protect your data.

Securing Kubernetes from the Linux kernel

Falco is designed to work in any Kubernetes cluster including Azure Kubernetes Service and deliver events to your existing security tools. This approach lets you include Falco results in Azure Sentinel and work with AI agents to help identify issues that might not be covered by Falco’s own rules.

Because Falco works at a low level and tracks service scaling, it is more like an observability tool than a traditional security monitor. Degioanni thinks of Falco as a question, “What does it mean to secure something that is so dynamic that changes so much?”

Working with Kubernetes as a microservice orchestrator adds complexity, too. Degioanni describes the challenge:

Users have 500 containers in a single, maybe 96-core machine. If you have to put something inside all of these and they go up and down, you know, so it’s absolutely unfeasible. So, what we do is… the Falco probe goes in the kernel of the operating system. So no matter how many containers you have in this 96-core machine, you only have one instrumentation. And when a container goes up and down, you don’t have to wait for your instrumentation to be active in that container because it’s already active in the underlying kernel.

The easiest way to install software in Azure Kubernetes Service is to use Helm. The Falco community maintains a Helm chart that installs Falco as a DaemonSet from the Falco GitHub repository. This approach allows you to automate deployment, using Azure DevOps or GitHub Actions. There’s really no point in thinking about manual deployments, as you’re going to be using Falco as part of an idempotent infrastructure, where each new deployment replaces the previous one.

Installing the Falco Helm chart is easy enough. Simply run the chart from its repository, creating a Falco namespace in your cluster, and setting up instances in your nodes. If you’re running Falco as a DaemonSet there should be one Falco pod per node, using your current container runtime.

Running Falco in Kubernetes hosts

If, instead of using Azure Kubernetes Service, you’re setting up your own Kubernetes environment on Azure, you can install Falco in the host system, isolating it from the Kubernetes instance it’s monitoring. Alerts are still managed inside Kubernetes, but a host-based install gives you a fallback in the event of a severe compromise. The Falco team provides installation instructions for different Linux versions, both Debian-based distributions like Ubuntu and Red Hat-based distros. There’s support for ARM as well as x64, so you can take advantage of Microsoft’s new ARM-based high-density Azure servers.

This table and values.yaml file list Falco’s configurable parameters. These are used to control how the service operates, for example managing the drivers used to host probes and the APIs used to deliver data to external services.

Another feature of the Falco Helm chart is its set of default rules. These will get you started, but as your Kubernetes applications grow and as you get more experience running cloud-native security, you can add your own custom rules in their own file is referred to from the installation chart.

Falco rules are written in YAML and can refer to additional rule sets. For example, you’re able to ensure that your application only uses defined ports or that it can spawn only specific, known processes. Anything that happens outside the default and custom rules will trigger an alert.

Feeding the security monitor

If you’re already using an Azure security tool like Sentinel, you’ll want to add Falcosidekick to your installation. This is an official extension that helps manage alerts, delivering them to Azure Event Hub and on to your SIEM (security information and event manager). Usefully the sidekick includes the ability to add custom fields to alerts, allowing you to add details of the Kubernetes cluster sending the alert. This is important when you’re monitoring multiple clusters, either running different applications in each cluster or geo-replicating applications across Azure regions.

Other tools let you plug into Azure Monitor managed service for Prometheus, which lets you use Grafana to build and run dashboards that deliver security information to your secops team.

Why use Falco instead of other Kubernetes security tools? The answer is simple: Falco is able to identify new attack patterns that may not be visible to other tools. Whether an attacker operates as a slow persistent threat or attempts a quick in-and-out exfiltration, understanding that an intrusion has occurred lets you quickly respond, either locking down access or starting more complex forensic actions, while blocking those new unexpected behaviors.

Stopping cloud-native zero days

By using eBPF probes to identify unfamiliar activities at a kernel level, Falco keeps you on top of possible zero-day exploits in your code. Rules can be written to alert you to activities that aren’t part of your application baseline and that aren’t covered by existing rule sets. Thus Falco gives you a catch-all that covers previously unseen actions—which might indicate bugs in your code or the operations of a threat actor.

However, there’s no point in deploying Falco on its own. Azure has its own security tools that work with Falco to provide a layered approach to securing your Kubernetes environment. Falco isn’t designed to work with .NET applications or to monitor Redis or PostgreSQL, so you will need to work with all of the available tools to protect your services.

Feeding a SIEM with all of the signals from across your application environment is key. You need to be able to work with Falco and all your other monitoring tools to understand what’s an attack and what’s a failure, and to tailor your responses appropriately. Now that Falco has graduated to become a full-fledged CNCF project, integrated in the Kubernetes platform stack, we can start to put those pieces together and build secure and reliable cloud-native applications, both on premises and in the cloud.