Exploring container security — Cilium way

Image generation: DALL.E

In my previous blog, we discussed about Berkeley Packet Filter (BPF ) technology. In this blog we will look in to a real life application of BPF.

Cilium is open source software for securing the network connectivity between application using BPF to bring API-aware network security filtering. Using BPF, a Linux kernel technology, Cilium provides simple and efficient way to define and enforce both network-layer and application-layer security policies based on container/pod identity.

Background

The shift to microservices in datacenter application development introduces new challenges, such as the need for API-aware security and policy enforcement, due to the highly volatile lifecycle of containers and the frequent churn of their IP addresses.

Traditional Linux security methods, like iptables, cannot support API-aware security and struggle to scale when IP addresses are frequently changing.

Cilium Functionality

Cilium provides an alternate to achieve above mention limitations by providing following features:

• Protect and Secure APIs transparently with its ability to secure modern application protocol such as REST/HTTP, gRPC and Kafka
• Secure service to service communication based on identities: Groups of application containers with the same identity share identical security policies.
• Secure access to and from external services: Support traditional CIDR based security policies
• Simple networking: A flat Layer 3 network capable of spanning multiple clusters connects all the application containers
• Load balancing: Distributed load balancing is implemented using BPF

Architecture

Cilium Architecture

Cilium Architecture Components

• Cilium Agent: Userspace daemon that interacts with the orchestration systems to setup networking and security for containers running on the local server
• Cilium CLI Client: Simple CLI client for communicating with the local Cilium agent
• Linux kernel BPF: Run at key points in the network stack to have visibility and control over all network traffic in/out of all containers
• Container Platform Network Plugin: Container platform has its own plugin model for how external networking platform integrate
• Apart form above components it also have following for cluster:

— Key-value store: Data share between Cilium Agents on different nodes

— Cilium Operator: Daemon for handling cluster management duties.

Now, lets look into the security implementation. Also, it is advisable to refer to my previous blog on Netfilter to refresh your knowledge on Linux Networking stack.

Scenario I: Endpoint to Endpoint (Local)

Before TCP ESTABLISHED

Endpoint to endpoint Datapath

Yellow boxes in above figure represent the Cilium implementation, lets look into various hooks that Cilium uses.

bpf_lxc:

TC HOOK @ source Endpoint:

• without L7 egress policy, it verify L3/L4 egress policy and get redirect identify of the target from bpf map and redirect the packet to the destination
• with L7 egress policy, it redirect to userspace proxy.

TC HOOK @ destination Endpoint:

• Without L7 ingress policy, it verify L3/L4 ingress policy
• with L7 policy, it redirect to userspace proxy

bpf_netdev

TC HOOK @ cilium_host of L7 egress/ingress policy: it source NAT for direct server return

Scenario II: Endpoint to Endpoint (Local)

TCP ESTABLISHED

bpf_sockops:

• It identify candidates sockets for accelerating, including all local node connections (endpoint to endpoint) and any connection to a Cilium proxy and check L3/L4 policy

bpf_redir:

• It is used for socket send/recv accelerating by redirect and it also check L3/L4 policy.

Scenario III: Egress from Endpoint

Egress from endpoint data path

• bpf_lxc (TC HOOK @ Endpoint): Verify L3/L4 egress policy, get redirect identity to target bpf map and redirect to cilium_host or TC HOOK@NIC if L3 encryption is enabled
• bpf_netdev (TC HOOK @ cilium_host): send to the world
• bpf_sockops & bpf_sockmap: When L7 policy in use, it redirect socket package to proxy directly
• bpf_network (TC HOOK @ NIC): It is used to encrypt the packet with xfrm*

Scenario IV: Ingress to Endpoint

Data path for Ingress to Endpoint

• bpf_xdp (XDP@NIC): It provides a set of prefilter rules used to filter traffic from the network for best performance
• bpf_lb (TC@NIC): As a load balancer, it performs a map lookup on the destination IP and optionally destination port for every packet and forward it to the matching endpoint.
• bpf_sockops & bpf_sockmap: When L7 policy in used, it redirect socket package from proxy directly

Summary

With eBPF, it make it more efficient than traditional Linux security methods. Cilium is based on identities which make it easy to scale. It can be extended with Prometheus to gain better visibility and can easily be integrated with other container platform like K8s.

Other projects the leverages BPF for container security

• Tracee: Its a lightweight, easy to use container and system tracing tool based on eBPF from Aqua Security (https://github.com/aquasecurity/tracee)
• AWS Firecracker: An open source virtualization technology provide serverless operational models, which uses Seccomp BPF to restrict system calls. (https://github.com/firecracker-microvm/firecracker)

References:

Component Overview - Cilium 1.17.0-dev documentation

https://github.com/cilium/cilium/tree/1c466d26ff0edfb5021d024f755d4d00bc744792/bpf