Linux Kernel Tracing

Description

Linux kernel provides building blocks for advanced debugging and tracing of applications as well as the kernel itself. Those mechanisms enable skilled developers to examine a running system and check for potential bugs and performance bottlenecks.

Skilled developers can use Linux kernel tracing mechanisms to explore and learn application and kernel behavior in a running environment and collect crucial events and statistics. Collected and analyzed information will help them understand various risks, bottlenecks and bugs in the system.

Target

This workshop is intended for groups of advanced Linux developers who would benefit from the knowledge of kernel and application instrumentation based on the Linux kernel features.

Contents

• Linux kernel instrumentation mechanisms
  • The ptrace debugger API
  • Event tracing
  • Function tracing
  • And more
• Collecting data
  • Kernel ring buffers
  • User-space consumers
• Tracing tools
  • Custom kernel modules
  • Tracer module generator – systemtap
  • DebugFS tracer – ftrace
  • Simple frontend for that – trace-cmd
  • perf tool
• eBPF subsystem
  • Bytecode language
  • Events, probes, profiling
  • Simple bpftrace tracing language
  • BCC eBPF compiler tools
• Case studies
  • Application to operating system interactions
  • System load characteristics
  • Process and thread scheduling
  • Hardware driver behavior
  • Real-time latency profiling