WIXLIB

eBPF: The next power tool of SRE’sMichael KehoeSr Staff Security Engineer

AgendaAn introduction & history of BPFCapability 2: NetworkingWhat is all the fuss about?Firewall, DDoS, Load-balancingHow to get started with eBPFCapability 3: SecurityWrite your first programContainer & LSM controlsCapability 1: Observability & TracingThe future of eBPF & SREHigh performance, high fidelity tracingWhere are we going2

Introduction

Introduction: Michael Kehoe Sr Staff Security Engineer - Confluent InfraSec/ CloudSec team Previously: Sr Staff SRE @ LinkedIn PhoneSat intern @ NASA Background in: Networks Microservices Traffic Engineering KV Databases Incident Management Twitter: @michaelkkehoe LinkedIn: linkedin.com/in/michaelkkkehoe Website: michael-kehoe.io4

5

An Introduction to eBPF

Put your hand up if you’ve used BPFbefore?7

Put your hand up if you’ve used tcpdumpbefore?8

What is cBPF? cBPF - Classic BPF Also known as “Linux Packet Filtering” BPF was first introduced in 1992 by Steven McCanne and Van Jacobson in BSD Implemented in Linux kernel 2.2 (Linux Socket Filtering) Originally used for network packet filtering & later, seccomp Works by: Filter expressions byte code interpreter Uses: Small, in-kernel VM, Register based, limited instructions9

What is cBPF?10

What is eBPF?11

“eBPF does to Linux what JavaScript does to HTML”Brendan GreggSr Performance Engineer, Netflix12

“eBPF is Linux’s new superpower”Gaurav GuptaSAP Labs13

“BPF is a highly flexible and efficient virtual machine-likeconstruct in the Linux kernel allowing to execute bytecodeat various hook points in a safe manner. It is used in anumber of Linux kernel subsystems, most prominentlynetworking, tracing and security (e.g. sandboxing).”Cilium14

What is eBPF? eBPF - extended Berkeley Packet Filter User defined, sandboxed bytecode executed by the kernel VM that implements a RISC-like assembly language in kernel space Multiple verification layers to ensure kernel safety Interactions between kernel/ user space are done through eBPF “maps” And blocking trace pipes eBPF does not allow loops* Kernel-like functionality without the FUD* Bounded loops in kernel 5.315

What is BPF-5May2017%20%281%29.pdf

bpf() system callbpf cmdInterface between user-space & eBPF VM17

eBPF Program Typesbpf prog typeDetermines the subset of kernel helperfunctions the program may callbpf contextThe program type will help determine theset of arguments given to a eBPF program18

eBPF Map TypeseBPF Maps Generic structure for storage of differentdata typesAllows sharing of data: Within an eBPF program Between kernel & user space19

eBPF HelperseBPF Helpers Specific functions to be run within aneBPF programVarious functionality Manipulating maps Debug functions Load data from packets .and moreCheck your kernel for focal/man7/bpf-helpers.7.html20

How to get started with eBPF

Where to get started with eBPF1. Run the most recent kernel possible2. Ensure that eBPF kernel configuration options are set to ‘y’3. Install bcctools (https://github.com/iovisor/bcc/)4. Start coding22

Where to get started with eBPFCONFIG BPF yCONFIG BPF SYSCALL y# [optional, for tc filters/ actions]CONFIG NET CLS BPF mCONFIG NET ACT BPF mCONFIG BPF JIT y# [for Linux kernel versions 5.7 and later]CONFIG BPF LSM y# [for Linux kernel versions 4.7 and later]CONFIG HAVE EBPF JIT y# [optional, for kprobes]CONFIG BPF EVENTS y# Need kernel headers through /sys/kernel/kheaders.tar.xzCONFIG IKHEADERS y23

How to get started with eBPF# CentOS/ Redhat sudo yum install bcc bcc-doc bcc-tools# Debian/ Ubuntu sudo apt-get install bpfcc-tools linux-headers- (uname -r)24

Where to get started with eBPF: Hello Worldfrom bcc import BPF# Kernel-Spaceprog “““int kprobe sys clone(void *ctx) {bpf trace printk("Hello, World!\\n");return 0;}”””# User-SpaceBPF(text prog).trace docs/tutorial bcc python developer.md25

Where to get started with eBPF: Hello Worldmichael@laptop: sudo python ebpf demo.pyb' Privileged Cont-3480 [005] d. 78819.733331: bpf trace printk: Hello, World!'b''b' WebExtensions-3801 [001] d. 78819.816553: bpf trace printk: Hello, World!'b''b' WebExtensions-3801 [001] d. 78819.822080: bpf trace printk: Hello, World!'b''b' WebExtensions-3801 [001] d. 78819.822308: bpf trace printk: Hello, World!'b''b' WebExtensions-3801 [001] d. 78819.822495: bpf trace printk: Hello, World!'26

Capability 1:Observability

eBPF ObservabilityK(ret)probes/ U(ret)probes Captures the entering (or exiting) of akprobe or uprobe Exceptionally useful for capturing: Disk operations Network connections Execution of programsUSDT’s Captures user statically definedtracepoints (USDT’s) in a program You can add tracepoints to your ownprogram and then debug it with eBPF28

eBPF ObservabilityTracepoints Allows you to instrument (pre-defined)tracepoints in kernel code.Perf Events Allows you instrument software andhardware performance eventsotherwise known as perf-events Can have higher performance thankprobes29

Observability: disksnoop.pyfrom bcc import BPFfrom bcc.utils import printbb BPF(text """#include uapi/linux/ptrace.h #include linux/blk-mq.h BPF HASH(start, struct request *);void trace start(struct pt regs *ctx, struct request *req) {// stash start timestamp by request ptru64 ts bpf ktime get ns();start.update(&req, &ts);}void trace completion(struct pt regs *ctx, struct request *req) {u64 *tsp, delta;tsp start.lookup(&req);if (tsp ! 0) {delta bpf ktime get ns() - *tsp;bpf trace printk("%d %x %d\\n", req- data len,req- cmd flags, delta / .py30

Observability: disksnoop.pyb.attach kprobe(event "blk mq start request", fn name "trace start")b.attach kprobe(event "blk account io done", fn name "trace completion")while 1:try:(task, pid, cpu, flags, ts, msg) b.trace fields()(bytes s, bflags s, us s) msg.split()if int(bflags s, 16):type s b"W"elif bytes s "0": # see blk fill rwbs() for logictype s b"M"else:type s b"R"ms float(int(us s, 10)) / 1000printb(b"%-18.9f %-2s %-7s %8.2f" % (ts, type s, bytes s, ms))except or/bcc/blob/master/examples/tracing/disksnoop.py

Observability: disksnoop.py 14730216458044.148117T BYTESW 4096W 4096W 4096W 4096R 4096R 4096R 4096R 4096R 4096R amples/tracing/disksnoop example.txt

Capability 2:Networking

eBPF NetworkingLoad balancingControl of socketsEasily load-balance/ forward millions of packetsper secondAdditional controls for sockets after they havebeen createdNetwork Filters/ DDoS protectionFlow dissectionEasily firewall/ filter millions of packets persecondWrite custom programs to perform network flowdissection for monitoring & accountingTraffic Control (tc)Prioritize/ monitor flows34

eBPF Networking Katran (Facebook load balancer) Cilium/ Hubble (Kubernetes network load-balancing/ firewall & more) Calico (Kubernetes CNI) Cloudflare edge infra (read their blog) s/networking https://blog.cloudflare.com/tag/ebpf/35

Capability 3:Security

eBPF Securitycgroup device Control/ monitor usage of host’s devicesby a cgroupcgroup sysctl Control/ monitor usage of host’s sysctl’sby a cgroup37

eBPF Securitycgroup skb Firewall/ network-filters for cgroupsLSM Instruments an LSM hook as a BPFprogram. It can be used to audit security eventsand implement MAC security policies inBPF.38

Security: LSM exampleimport osimport sysimport timefrom bcc import BPF, libbccsrc """#include linux/fs.h #include uapi/asm-generic/errno-base.h LSM PROBE(file open, struct file *file) {bpf trace printk("LSM hook: file open\\n");u32 pid bpf get current pid tgid();if (pid ! 1) {bpf trace printk("LSM hook: file open: Denied\\n");return -EPERM;}bpf trace printk("LSM hook: file open: Allowed\\n");return 0;}"""Ref: html39

Security: LSM exampleb BPF(text src)fn b.load func("file open", BPF.LSM)try:while 1:time.sleep(0.5)print(b.trace fields())# Extra logging logicexcept KeyboardInterrupt:sys.exit()Ref: html40

The future of eBPF & SRE

The future of eBPF & SREObservability Allows you to troubleshootlow-level issues withoutworrying aboutperformance Never have to usestrace againNetworking Real-life examples inKubernetes/ Cilium Hyperscale for everyone: Firewalls Load-balancing WAFsSecurity Deep integration withLSM’s for rich runtimesecurity data Cgroup protections: Devices sysctl’s Network Traffic Opens up new possibilitiesto optimize user-ownedsoftware and locate bugs42

The future of eBPF & SRE: Words of caution Despite the performance of eBPF, you can still harm your system Know your performance boundaries/ limitations Be wary of OS/ kernel compatibility CentOS/ Redhat often backport to older kernels You will need to think about your deployment strategies (hint: look at CO-RE) Running programs via systemd is an option While eBPF is kernel-safe, you still need to thoroughly test before production43

Resources https://github.com/michael-kehoe/bpf-workshop https://ebpf.io/ https://docs.cilium.io/en/stable/bpf/ https://github.com/iovisor/bcc https://github.com/aquasecurity/tracee Linux Observability with BPF (Book) BPF Performance Tools (Book)44

Q&A

Network Filters/ DDoS protection Prioritize/ monitor flows Easily load-balance/ forward millions of packets per second Load balancing Traffic Control (tc) Additional controls for sockets after they have been created Control of sockets Write custom programs to perform network flow dissection for monitoring & accounting Flow dissection