Inspektor Gadget is a collection of tools (or gadgets) to debug and inspect Kubernetes resources and applications. It manages the packaging, deployment and execution of eBPF programs in a Kubernetes cluster, including many based on BCC tools, as well as some developed specifically for use in Inspektor Gadget. It automatically maps low-level kernel primitives to high-level Kubernetes resources, making it easier and quicker to find the relevant information.

The Gadgets

Inspektor Gadget tools are known as gadgets. You can deploy one, two or many gadgets.

Explore the following documentation to find out which tools can help you in your investigations.

• advise:
  • network-policy
  • seccomp-profile
• audit:
  • seccomp
• profile:
  • block-io
  • cpu
  • tcprtt
• snapshot:
  • process
  • socket
• top:
  • block-io
  • ebpf
  • file
  • tcp
• trace:
  • bind
  • capabilities
  • dns
  • exec
  • fsslower
  • mount
  • oomkill
  • open
  • signal
  • sni
  • tcp
  • tcpconnect
  • tcpdrop
  • tcpretrans
• script
• traceloop

Installation

Install Inspektor Gadget (client-side):

Use krew plugin manager to install:

$ kubectl krew install gadget

Install Inspektor Gadget on Kubernetes:

$ kubectl gadget deploy

Read the detailed install instructions to find more information.

How to use

kubectl gadget --help will provide you the list of supported commands and their flags.

$ kubectl gadget --help
Collection of gadgets for Kubernetes developers

Usage:
  kubectl-gadget [command]

Available Commands:
  advise      Recommend system configurations based on collected information
  audit       Audit a subsystem
  completion  Generate the autocompletion script for the specified shell
  deploy      Deploy Inspektor Gadget on the cluster
  help        Help about any command
  profile     Profile different subsystems
  prometheus  Expose metrics using prometheus
  run         Run a containerized gadget (experimental)
  script      Run a bpftrace-compatible scripts
  snapshot    Take a snapshot of a subsystem and print it
  sync        Synchronize gadget information with server
  top         Gather, sort and periodically report events according to a given criteria
  trace       Trace and print system events
  traceloop   Get strace-like logs of a container from the past
  undeploy    Undeploy Inspektor Gadget from cluster
  version     Show version

...

You can then get help for each subcommand:

$ kubectl gadget advise --help
Recommend system configurations based on collected information

Usage:
  kubectl-gadget advise [command]

Available Commands:
  network-policy  Generate network policies based on recorded network activity
  seccomp-profile Generate seccomp profiles based on recorded syscalls activity

...
$ kubectl gadget audit --help
Audit a subsystem

Usage:
  kubectl-gadget audit [command]

Available Commands:
  seccomp     Audit syscalls according to the seccomp profile

...
$ kubectl gadget profile --help
Profile different subsystems

Usage:
  kubectl-gadget profile [command]

Available Commands:
  block-io    Analyze block I/O performance through a latency distribution
  cpu         Analyze CPU performance by sampling stack traces
  tcprtt      Analyze TCP connections through an Round-Trip Time (RTT) distribution

...
$ kubectl gadget snapshot --help
Take a snapshot of a subsystem and print it

Usage:
  kubectl-gadget snapshot [command]

Available Commands:
  process     Gather information about running processes
  socket      Gather information about TCP and UDP sockets

...
$ kubectl gadget top --help
Gather, sort and periodically report events according to a given criteria

Usage:
  kubectl-gadget top [command]

Available Commands:
  block-io    Periodically report block device I/O activity
  ebpf        Periodically report ebpf runtime stats
  file        Periodically report read/write activity by file
  tcp         Periodically report TCP activity

...
$ kubectl gadget trace --help
Trace and print system events

Usage:
  kubectl-gadget trace [command]

Available Commands:
  bind         Trace socket bindings
  capabilities Trace security capability checks
  dns          Trace DNS requests
  exec         Trace new processes
  fsslower     Trace open, read, write and fsync operations slower than a threshold
  mount        Trace mount and umount system calls
  network      Trace network streams
  oomkill      Trace when OOM killer is triggered and kills a process
  open         Trace open system calls
  signal       Trace signals received by processes
  sni          Trace Server Name Indication (SNI) from TLS requests
  tcp          Trace TCP connect, accept and close
  tcpconnect   Trace connect system calls
  tcpdrop      Trace TCP kernel-dropped packets/segments
  tcpretrans   Trace TCP retransmissions

...

How does it work?

Inspektor Gadget is deployed to each node as a privileged DaemonSet. It uses in-kernel eBPF helper programs to monitor events mainly related to syscalls from userspace programs in a pod. The eBPF programs are run by the kernel and gather the log data. Inspektor Gadget's userspace utilities fetch the log data from ring buffers and display it. What eBPF programs are and how Inspektor Gadget uses them is briefly explained in the architecture document.

ig

Inspektor Gadget can also be used without Kubernetes to trace containers with the ig tool.

Kernel requirements

The different gadgets shipped with Inspektor Gadget use a variety of eBPF capabilities. The capabilities available depend on the version and configuration of the kernel running in the node. To be able to run all the gadgets, you'll need to have at least 5.10 with BTF enabled.

See requirements for a detailed list of the requirements per gadget.

Code examples

There are some examples in this folder showing the usage of the Golang packages provided by Inspektor Gadget. These examples are designed for developers that want to use the Golang packages exposed by Inspektor Gadget directly. End-users do not need this and can use kubectl-gadget or ig directly.

Contributing

Contributions are welcome, see CONTRIBUTING.

Community Meeting

We hold community meetings every other Thursday at 15:30 UTC, 7:30 PST, 16:30 CET in this link, check the calendar to have the full schedule of next meetings. Please add any topic you want to discuss to our meeting notes document.

Slack

Join the discussions on the #inspektor-gadget channel in the Kubernetes Slack.

Talks

• Collecting Low-Level Metrics with eBPF, KubeCon + CloudNativeCon North America 2023 (video, slides)
• A (re)introduction of Inspektor Gadget: A Containerized Framework for eBPF Systems Inspection, Cloud Native Rejekts Chicago - November 2023 (video)
• Gaining Linux insights with Inspektor Gadget, an eBPF tool and systems inspection framework, All Systems Go - September 2023 (video)
• Overcoming the Challenges of Debugging Containers, Container Days Hamburg - September 2023 (video)
• Using the EBPF Superpowers To Generate Kubernetes Security Policies, KubeCon + CloudNativeCon North America 2022 (video, slides)
• Debug Your Clusters with eBPF-Powered Tools, Cloud Native eBPF Day North America 2022 (video, slides)
• Who Needs an API Server to Debug a Kubernetes Cluster?, Cloud Native eBPF Day North America 2022 (video, slides)
• Inspektor Gadget, introduction and demos, eCHO Livestream - September 2021 (video)
• OpenShift Commons Briefing: Unleash eBPF Superpowers with Kubectl Gadget, Openshift Commons 2020 (video)
• Tutorial: Understanding What Happens Inside Kubernetes Clusters Using BPF Tools, Open Source Summit EU 2020 (video)
• Inspektor Gadget and traceloop: Tracing containers syscalls using BPF, FOSDEM 2020 (video, slides)
• Traceloop for systemd and Kubernetes + Inspektor Gadget, All Systems Go 2019 (video)

Thanks

• BPF Compiler Collection (BCC): some of the gadgets are based on BCC tools.
• kubectl-trace: the Inspektor Gadget architecture was inspired from kubectl-trace.
• cilium/ebpf: the gadget tracer manager and some other gadgets use the cilium/ebpf library.

License

The Inspektor Gadget user space components are licensed under the Apache License, Version 2.0. The BPF code templates are licensed under the General Public License, Version 2.0, with the Linux-syscall-note.