Debugging

What's the difference between command and command | cat? There shouldn't be any, right? The first prints the output directly, and the latter prints output via cat, but they should have the same effect. Not so: the latter can cause the command to get stuck indefinitely. This has given a particular CI pipeline of mine quite some headache.

To learn why this happens, and how we can mitigate this problem, we need to dive into the arcane magic that is Unix process management. Join me on this journey.

One of the projects I'm working on has a CI/CD pipeline that builds Docker images. The Dockerfile runs yarn install, then yarn build. The latter runs the TypeScript compiler tsc. Everything was working fine, but one day the build failed with the following error:

tsc: command not found

But TypeScript is still part of package.json. Nobody touched package.json or yarn.lock recently. Nobody could reproduce the problem locally with Docker: it only happened in the CI/CD pipeline. What is going on? We needed to debug the Docker build on the CI/CD server.

Linux has two well-known tracing tools:

• strace allows you to see what system calls are being made.
• ltrace allows you to see what dynamic library calls are being made.

Though useful, these tools are limited. What if you want to trace what happens inside a system call or library call? What if you want to do more than just logging calls, e.g. you want to compile statistics on certain behavior? What if you want to trace multiple processes and correlate data from multiple sources?

In 2019, there's finally a decent answer to that on Linux: bpftrace, based on eBPF technology. Bpftrace allows you to write small programs that execute whenever an event occurs.

This article shows you how to setup bpftrace and teaches you its basic usage. I'll also give an overview of how the tracing ecosystem looks like (e.g. "what's eBPF?") and how it came to be what it is today.