Most software engineers view CPU utilization as a “gas tank”: if the OS says a core is at 95% utilization, we assume it is doing 95% useful work. This is a microarchitectural illusion.
The OS scheduler calculates CPU percentage purely as time residency on a core. The OS has no idea whether the core is executing useful instructions or stalling indefinitely.
While writing and optimizing the native data-collection loop of our new open-source hardware topology TUI, swifttopology, we decided to go below standard /proc scraping. By binding directly to physical CPU Model-Specific Registers (MSRs) via raw perf_event_open system calls and eBPF maps, we began measuring actual instructions and cycles retired at the thread level.
What we found on our bare-metal Intel Xeon test node exposed a massive diagnostic gap in standard Linux monitoring tools.
1. The Core Abstraction: Why the OS Lies #
When you run top or look at your APM dashboards, CPU utilization is calculated over a sampling interval (e.g., 1 second). If a thread was assigned to a core for 950 milliseconds out of that 1,000ms window, the OS reports 95% CPU utilization.
But what was the CPU actually doing during those 950 milliseconds?
A CPU core is a highly complex out-of-order execution engine. It can spend clock cycles in two primary states:
- Retiring Cycles: The execution units are successfully executing and retiring instructions (productive work).
- Stalled Cycles: The execution pipeline is completely frozen, waiting for data to arrive from the cache hierarchy or system memory (DRAM), or waiting for a shared hardware resource to free up.
To the operating system, a CPU core retiring 4 instructions per cycle and a CPU core stalled waiting 200 nanoseconds for a memory line look identical. Both are marked as 100% busy.
To unmask what is actually happening on the silicon, we must measure Instructions Per Cycle (IPC):
An IPC of 1.0 means the CPU retires one instruction per clock cycle on average. Modern superscalar x86_64 cores can achieve an IPC of 2.0 to 3.0 under highly optimized, pipelined workloads. An IPC below 0.5 indicates the CPU is heavily stalled, spending the majority of its cycles doing absolutely nothing.
2. A Tale of Two Threads: The Telemetry #
We ran our compiled eBPF dynamic telemetry pipeline on a physical bare-metal server on Vultr and queried the raw thread execution data. We isolated two threads running concurrently on the same system. To standard Linux utilities, both looked busy and healthy. The microarchitectural counters told a completely different story.
┌────────────────────────────────────────────────────────────────────────┐
│ THE SILICON REALITY │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ kworker/4:0 ──────────────────────────────► [ High L3 Misses ] │
│ (28.6% CPU) [ IPC: 2.22 (Excellent)]│
│ Prefetchers hiding delay│
│ │
│ tokio-rt-worker ──────────────────────────► [ Low L3 Misses ] │
│ (95.5% CPU) [ IPC: 0.076 (Stalled) ]│
│ Spinning on user-space │
│ │
└────────────────────────────────────────────────────────────────────────┘Case Study A: The Highly Optimized Kernel Worker (kworker/4:0)
#
- CPU Utilization:
28.68% - Instructions Per Cycle (IPC):
2.22 - L3/Last Level Cache (LLC) Misses:
19,799.85/ second
An IPC of 2.22 on an Intel Xeon core is exceptional. It means the CPU is retiring more than 2 instructions on almost every single clock cycle. What is fascinating is that it maintains this high execution efficiency despite missing the L3 cache nearly 20,000 times per second.
Under normal circumstances, a single L3 cache miss forces the core to query physical DRAM, taking 200 to 300 CPU cycles (latency). If a thread misses the L3 cache 20,000 times a second, those latency stalls should freeze the pipeline, dragging the average IPC down below 0.5.
The Systems Reality: This indicates that the kernel worker is executing highly predictable, sequential loops (such as copying flat memory buffers or processing network packet arrays).
Because the memory access pattern is linear and predictable, the CPU’s physical Stream and Spatial Prefetchers anticipate the memory offsets and issue DRAM read requests before the instruction execution stream actually requests them. By the time the instruction is executed, the data is already sitting in the L1/L2 cache. The memory latency is completely hidden, allowing the execution pipeline to remain full.
Case Study B: The Spinning Runtime Worker (tokio-rt-worker)
#
- CPU Utilization:
95.50% - Instructions Per Cycle (IPC):
0.076 - L3/Last Level Cache (LLC) Misses:
3,619.12/ second
Here, the operating system reports that a Tokio runtime worker thread is saturating a CPU core at 95.50% utilization. But its IPC is 0.076—meaning that for every 1,000 CPU cycles allocated to this thread, only 76 instructions are actually retired. The execution pipeline is stalled 92.4% of the time.
The Systems Reality:
- Is it waiting on memory? No. An L3 miss rate of
3,619/sec is negligible. A modern memory bus can handle millions of misses per second. This minor cache miss rate cannot physically explain a 92% pipeline stall rate. - Is the thread blocked or sleeping? No. If the thread had called a blocking system call (like waiting on a standard OS mutex or sleeping), the kernel would put the thread into a sleeping state and context-switch it out. Its CPU utilization would drop to 0%.
Because it is occupying 95% of a core, is not waiting for memory (low L3 misses), and is not retiring useful instructions (low IPC), this thread is spinning in user-space (busy-waiting on an atomic flag, a lock, or an unyielded asynchronous event loop).
When a thread spins in a tight loop waiting for an atomic flag to change, the instruction loop is tiny and resides entirely in the CPU’s L1 Instruction Cache, resulting in zero cache misses. However, because there is no PAUSE instruction or yielding, the CPU’s Out-of-Order execution engine speculatively executes hundreds of iterations of the loop ahead. When the physical pipeline resources (like Reorder Buffers or Load/Store queues) fill up, the CPU hits Resource Starvation Stalls. The core runs at 100% clock cycles, but retires virtually zero useful instructions.
To the outside observer, it looks like a highly active compute-bound thread. In reality, it is burning electricity (Joules) while doing absolutely nothing.
3. The Implementation: Thread-Level PMC Attribution #
To capture these metrics without introducing a heavy “APM tax” (which would defeat the purpose of an optimization tool), we cannot use standard user-space sampling or run heavy scripts at runtime. We must map physical CPU registers directly to our data-collection loop using eBPF.
However, physical CPU Performance Monitoring Counters (PMCs) are global registers per CPU core. They increment continuously as different threads execute on a core. If Thread A runs on Core 0, then Thread B runs, and then Thread A runs again, the hardware counter has been incrementing the entire time.
If we just write the absolute counter value to our thread records, Thread A will be falsely credited with the millions of cycles executed by Thread B while Thread A was sleeping!
To solve this, we implemented a Delta-Accumulation State Machine inside our eBPF scheduler hooks:
// bpf/main.bpf.c
// 1. Define the Hardware PMC maps (mapped per-CPU)
struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
} cycles SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
} instructions SEC(".maps");
// Cache map to store PMC baselines per CPU core on sched_in
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 256);
__type(key, u32);
__type(value, struct pmu_sample);
} pmu_start_values SEC(".maps");
SEC("tp/sched/sched_switch")
int handle_sched_switch(struct trace_event_raw_sched_switch *ctx) {
u64 now = bpf_ktime_get_ns();
u32 prev_tid = BPF_CORE_READ(ctx, prev_pid);
u32 next_tid = BPF_CORE_READ(ctx, next_pid);
u32 cpu_id = bpf_get_smp_processor_id();
// Read current CPU hardware registers
s64 current_cycles = bpf_perf_event_read(&cycles, cpu_id);
s64 current_instructions = bpf_perf_event_read(&instructions, cpu_id);
// PART A: The thread leaving the CPU (Exec ends)
struct thread_metrics *m_prev = bpf_map_lookup_elem(&metrics_map, &prev_tid);
if (m_prev && m_prev->last_active > 0) {
// Retrieve PMC baselines recorded when this specific thread entered this CPU core
struct pmu_sample *pmu_start = bpf_map_lookup_elem(&pmu_start_values, &cpu_id);
if (pmu_start && current_cycles > 0 && current_instructions > 0) {
// Calculate absolute execution delta strictly belonging to this timeslice
u64 delta_cycles = (u64)current_cycles - pmu_start->cycles;
u64 delta_inst = (u64)current_instructions - pmu_start->instructions;
if ((s64)delta_cycles >= 0 && (s64)delta_inst >= 0) {
m_prev->cycles += delta_cycles;
m_prev->instructions += delta_inst;
}
}
}
// PART B: The thread entering the CPU (Wait ends, Exec starts)
struct thread_metrics *m_next = get_or_create_metrics(next_tid);
if (m_next) {
m_next->last_active = now;
// Cache the current physical PMC registers as the start baseline for the entering thread
struct pmu_sample *pmu_start = bpf_map_lookup_elem(&pmu_start_values, &cpu_id);
if (!pmu_start) {
struct pmu_sample new_start = { .cycles = (u64)current_cycles, .instructions = (u64)current_instructions };
bpf_map_update_elem(&pmu_start_values, &cpu_id, &new_start, BPF_ANY);
} else {
pmu_start->cycles = (u64)current_cycles;
pmu_start->instructions = (u64)current_instructions;
}
}
return 0;
}By calculating the delta ($current - baseline$) during context switches and accumulating it, we ensure that a thread’s hardware counters only increment when that thread is actually executing on the CPU core, providing perfect, unpolluted thread-level PMC diagnostics.
4. Conclusion & Play with the Silicon #
If you are investigating performance regressions, profiling high-throughput asynchronous runtimes, or optimizing database cores, you must stop relying on superficial OS metrics. You must look at how the software is executing on physical silicon.
We have packaged this native, zero-dependency topology and performance collection mapping directly into our open-source, lightweight TUI tool: swifttopology.
You can download the single static binary, run it natively on your physical x86_64 or ARM servers, and see your CPU core boundaries and real-time execution classifications unmasked in seconds:
# Download and run swifttopology instantly
curl -L https://github.com/swiftlogicsystems/swifttopology/releases/latest/download/swifttopology -o swifttopology && chmod +x swifttopology
sudo ./swifttopologyStop guessing why your cores are busy. Let the silicon tell you.