Concurrency bugs and debugging

Throughout the series we stacked up tools one at a time. Starting threads (part 1), locking (part 2), signalling (part 3), passing results (part 4), sharing without locks (part 5). The final part is about what happens when those tools are used wrongly, and how to catch bugs that are hard even to reproduce. Concurrency bugs usually fire only “sometimes”, so knowing the preventive measures and the tools is itself the skill.

Deadlock: waiting for each other forever

A deadlock is a state in which two or more threads each wait for a resource the other holds and nobody makes progress. The most common form is taking two mutexes in different orders.

std::mutex m1, m2;

// thread A
void a() {
    std::lock_guard<std::mutex> l1(m1);   // takes m1
    std::lock_guard<std::mutex> l2(m2);   // waits for m2
}

// thread B
void b() {
    std::lock_guard<std::mutex> l1(m2);   // takes m2
    std::lock_guard<std::mutex> l2(m1);   // waits for m1
}

A holds m1 and waits for m2; if at that moment B holds m2 and waits for m1, the two wait for each other to let go, forever. The program hangs without emitting any error.

Fix 1: always lock in the same order. If every thread respects a global order of “m1 first, then m2”, this circular wait cannot arise in the first place. This is the most basic and most powerful rule.

Fix 2: lock them together in one go. When you need several mutexes at once, use std::scoped_lock (C++17) from part two. It locks them all at once using a deadlock-avoiding algorithm internally.

void a() {
    std::scoped_lock lock(m1, m2);   // both at once, without deadlock
}
void b() {
    std::scoped_lock lock(m2, m1);   // safe even in a different order
}

Fix 3: minimise how long you hold a lock. Do not take another lock, invoke a callback, or do anything long-running inside a critical section. Calling an external callback while holding a lock is especially risky: that callback may take another lock and create an unexpected cycle.

Data race: silent UB

A data race was defined in part five. Two accesses with no happens-before relation touch the same location, at least one of them a write — that is a data race, and it is undefined behaviour.

Unlike a deadlock, a data race does not hang. Most of the time it just looks like it works. Then, on a particular timing, a particular optimisation level, or particular hardware, a value gets corrupted, or it crashes, or — worse — it produces a wrong answer only occasionally. A large share of “works on my machine” and “works in debug but not in release” bugs are exactly this.

The key point is do not try to catch it with your eyes. Data races are easy to miss in code review, and their non-deterministic reproduction makes them hard to chase with a debugger. Use a dedicated tool instead.

ThreadSanitizer: the tool for catching races

ThreadSanitizer (TSan) is a dynamic analysis tool built into GCC and Clang. It tracks memory accesses at runtime and catches data races. One compile flag turns it on.

g++ -fsanitize=thread -g program.cpp -o program
./program

When a race occurs, TSan reports which two threads collided, on which variable, and at which source locations, complete with stack traces. It pinpoints in seconds a bug that would take days by eye.

Two cautions. TSan only catches races on paths that actually executed. If a code path is not exercised during the test, it is missed. So it is most effective run alongside tests that stress the concurrent code. TSan also slows execution considerably (5-15x) and uses more memory, so run it in CI and testing rather than permanently in production.

On Linux, Helgrind (a Valgrind tool) does something similar. It works without recompiling, but is far slower than TSan. If you can recompile, TSan is usually better.

False sharing: the invisible performance trap

This is not a bug — the results are correct. But it is a trap that slows you down for no apparent reason, so it is worth knowing.

CPUs handle memory not in bytes but in cache lines (typically 64 bytes). Even if two threads touch different variables, if those two variables sit in the same cache line, every write by one thread invalidates the other thread’s cache. There is no logical sharing, yet at the hardware level they share a cache line and obstruct each other. This is false sharing.

struct Counters {
    std::atomic<int> a;   // written by thread 1
    std::atomic<int> b;   // written by thread 2
};                        // if a and b share a cache line, they obstruct each other

When two counters sit adjacent and fall into the same 64-byte line, performance collapses under cache invalidation traffic even though the two threads are busily incrementing different counters. The fix is to separate each variable onto its own cache line. C++17 provides a constant for this.

#include <new>

struct Counters {
    alignas(std::hardware_destructive_interference_size)
        std::atomic<int> a;
    alignas(std::hardware_destructive_interference_size)
        std::atomic<int> b;
};   // a and b land on different cache lines

False sharing usually surfaces when you dig into “why didn’t adding threads make this faster?” with a profiler. Atomic counters or per-thread data packed tightly into an array is a place to suspect.

Handling bugs that are hard to reproduce

The defining characteristic of concurrency bugs is non-determinism. With the same input they pass or fail depending on execution order. There are a few practical strategies.

Apply pressure. Raise the thread count, raise the iteration count, run for a long time across many cores. A rare race shows itself over millions of repetitions. Combining a stress test with TSan is especially powerful.

Simplify. Build the minimal code that reproduces the bug. Cut down to two threads and strip out unrelated logic, and the heart of the problem becomes visible.

Do not lean on logging. Adding a printf or a log statement often shifts the timing and makes the bug disappear (the so-called heisenbug). Prefer tools like TSan over logs, and above all a minimal case that reproduces deterministically.

Prevent it by design. The surest debugging is not creating the bug at all. Reduce shared mutable state, document lock ordering, and where possible avoid sharing altogether using the higher-level tools from this series (async/future, message queues). Data that is not shared cannot race.

Closing the series

Across six parts we covered the practical core of C++ concurrency. The flow again, briefly:

Start threads (part 1), lock shared resources with a mutex (part 2), wait and wake with a condition_variable (part 3), pass results with future/async (part 4), share without locks using atomics (part 5), and in this part, what to do when all of it goes wrong.

A few principles ran through it. Shared mutable state is the root of every problem — the less of it, the safer. Automate cleanup with RAII — lock_guard, jthread, and future all embody this. When in doubt, pick the stronger option — a seq_cst atomic, a mutex, seq_cst ordering may be slow, but they are not wrong. And trust tools, not your eyes — TSan catches what people miss.

Concurrency is hard. But most of what makes it hard is the non-determinism of “only wrong sometimes”, and that non-determinism can be tamed with the right tools and discipline. I hope this series has been a starting point.