---
title: "Why the Slow Operator Is Slow: Attributing GPU Hardware Counters to PyTorch Operators"
slug: why-the-slow-operator-is-slow-gpu-counters-pytorch-operators
description: "Wall time tells you where, not why. Operator Profiler attributes the full GPU hardware counter set to PyTorch operators, with evidence behind every number."
author: "Yotta Labs"
date: 2026-08-18
categories: ["Research"]
canonical: https://www.yottalabs.ai/post/why-the-slow-operator-is-slow-gpu-counters-pytorch-operators
---

# Why the Slow Operator Is Slow: Attributing GPU Hardware Counters to PyTorch Operators

![](https://cdn.sanity.io/images/wy75wyma/production/cabcb68877d6d47519672a7ee5c2086c18eebc70-1656x866.png)

*Attributing GPU hardware counters to PyTorch operators.*

## Wall time tells you where, not why

A GPU workload's throughput is usually dominated by a single bottleneck class, bandwidth saturation, Tensor Core under-utilization, register-pressure-limited occupancy and launch-width starvation among them. Each of those is measurable, as is every operator's wall time; they are simply measured by different tools, in different passes.

Knowing that aten::mm consumed 6.4 ms tells you where the time went. It does not tell you whether that GEMM was starved for bandwidth, capped by occupancy, or dispatched onto the FP32 SIMT path with its Tensor Cores idle. **Three diagnoses, three unrelated fixes, one identical number.** The roofline taxonomy applies in principle; it just isn't computable at operator granularity with the tools as shipped.

Each tool holds one piece. nsys (Nsight Systems) records the kernel timeline and knows nothing about PyTorch. ncu (Nsight Compute) reads the full hardware counter set per kernel and knows even less, it replays kernels outside their original context and labels them by kernel name alone. torch.profiler knows the operators and, by default, reads no counters at all. None of them is wrong; the information is simply split three ways, with nothing in the stack putting it back together. Operator Profiler, the last row below, is what putting it back together produces.

<!-- unsupported block: table -->

What each tool actually produces. The compiler stack doesn't close the gap either: Inductor fuses from heuristics, TVM and Halide auto-tune against latency without ever naming the bottleneck class that produced it, TensorRT and ONNX Runtime fuse behind interfaces you can't interrogate. Classic PGO solved the analogous problem for CPUs thirty years ago by feeding branch-frequency profiles back into the compiler.

## What PyTorch already gives you, and what it doesn't

PyTorch has shipped a route to per-operator counters for years, and any honest account of this gap starts there. Since April 2022, torch.profiler can program NVIDIA's CUPTI Range Profiler through its experimental config:

```text
torch.profiler._ExperimentalConfig(
    profiler_metrics=["kineto_tensor_core_insts",
                      "dram__bytes_read.sum", "dram__bytes_write.sum"],
    profiler_measure_per_kernel=True)

```

Counters land in the same Kineto trace as the operator dispatches, and Holistic Trace Analysis closes the loop: get_cupti_counter_data_with_operators() returns one row per kernel carrying the full nested op_stack that launched it, plus a column per requested counter. If your question is answerable from a handful of raw counters, the documented example builds FLOP counts for a roofline, this is the shortest path there is, and it is built in. Our own highest-confidence tier uses the same CUPTI correlation mechanism. Kernel-to-operator correlation is not the contribution here.

Four things that path does not give you.

**You name the counters; you derive the diagnosis yourself.** The Range Profiler takes raw PerfWorks metric names. ncu ships the layer above them: derived, per-architecture metrics with the denominators already worked out, achieved occupancy, SM and DRAM throughput as a share of peak, Tensor Core pipe activity, cache hit rates, registers per thread, spill counts. That vocabulary is the bottleneck taxonomy. On the raw path you assemble and maintain it yourself, per architecture, and the names move: seven of the twenty we collect were renamed in Blackwell silicon.

**It only reaches runs you can instrument in-process.** Collection happens inside your torch.profiler context. A serving stack you don't own, a capture someone else took, a metric that needs ncu's multi-pass replay, none of those have a route back to operators.

**Nothing records how a mapping was obtained, or what failed to map.** A kernel has an op_stack or it doesn't appear. There is no tier to read and no accounting of the kernels that mapped to nothing, which, as the LSTM case below shows, is sometimes the entire diagnosis.

**You are already running two passes anyway.** nsys and torch.profiler both register CUPTI subscribers and cannot coexist in one process. The moment you want the timeline alongside the counters, you have two captures on incompatible clocks and nothing joining them. (The feature is also still experimental, riding CUPTI APIs that keep moving, the least interesting of the four objections, and the one most likely to be fixed.)

So the real gap is narrower than "nobody can do this": **nsys's timeline, ncu's full counter set, and PyTorch operator identity, reconstructed after the fact from artifacts three independent tools wrote.** Which returns us to the obvious bridge, run both, join on kernel name, and why it fails. Names aren't identifiers: one GEMM across 12 transformer blocks yields 12 indistinguishable rows. Timestamps aren't a tiebreaker: the two tools count on clocks that diverge **10–100 μs per kernel** and drift further apart across the trace. And a correctly joined kernel still carries no operator identity, triton_poi_fused_relu_addmm_0 is no single aten:: dispatch by construction. Bridging by hand does fix all three, at O(operators) of manual work, repeated for every model variant.

## Every number should say where it came from

One commitment drives everything below: **every number in the profile says where it came from.** Not just the value, the key it was joined on, the evidence that gave it an operator, how much that evidence is worth, and, when nothing matched, an explicit record of that.

That sounds like bookkeeping. It is the difference between a profile you can act on and a profile you have to trust. Reconstructing attribution across two tools is an act of inference, and inference you can't audit is indistinguishable from a plausible wrong answer. Four consequences follow, and together they are the whole system:

1. a join key both tools genuinely agree on, invocation order, never a timestamp;
2. an evidence tier stamped on every attribution, recorded rather than averaged away;
3. unattributed as a reported outcome, not a dropped row;
4. measurement conditions held fixed, so two profiles are comparable at all.

The payoff shows up twice in the results below: once when the tier, not the counter, turned out to be the whole diagnosis, and once when the profile priced the overhead its own optimization had introduced.

![](https://cdn.sanity.io/images/wy75wyma/production/97c08898f9dd337923dcfccead3b603deda48edc-1116x424.png)

## Step 1: Match kernels by execution order, not by timestamp

Because ncu collects counters by replaying the workload, its timestamps belong to a different run on a different clock, which is exactly why a timestamp join fails. The ordering, though, survives the replay: ncu re-executes kernels in the order they appear in the nsys timeline. The i-th invocation of kernel K in the trace is the i-th row for K in the ncu CSV. **Position is a key; time is not.**

![](https://cdn.sanity.io/images/wy75wyma/production/b13c6deb7ff58b6c21cf84f560fb6e6c68aeead7-1116x456.png)

```text
counter ← {}                          # invocation count per kernel name
for k in nsys_kernels:                # sorted by start_ns, per stream
    i ← counter[k.name]
    if i ≥ len(ncu_rows[k.name]):
        warn(k.name, i); continue     # count mismatch → skip, don't guess
    k.metrics ← ncu_rows[k.name][i]
    counter[k.name] += 1

```

The precondition is workload determinism: identical kernel names, counts and execution order across both runs. Conditional control flow, dynamic dispatch and allocator-emitted workspace kernels can all violate it without producing a shape error. So the invariant gets checked, invocation counts are validated before any counter is assigned, and a mismatch raises. **Skipping a kernel is a hole in coverage; misattributing one is a corrupted profile.**

## Step 2: Three ways to find a kernel's operator, ranked by trust

Counter-enriched kernels still need names. Three paths supply them, tested in descending confidence order, with the winning path written into the output, so a record carries not just which operator it was attributed to, but on what evidence.

![](https://cdn.sanity.io/images/wy75wyma/production/f484d0c7d07e34ca1c2300e1f798e1bc717997d2-1116x582.png)

**HIGH is a causal record, not an inference.** With Kineto active, CUPTI assigns a correlation ID at cuLaunchKernel and propagates it through the driver stack to every kernel that dispatch triggers. The resulting {(kernel name, i) → aten::op} map needs no timestamp comparison and no pattern matching on kernel names. It's a record of what caused what.

**NVTX enclosure queries the host clock, deliberately.** emit_nvtx() brackets each aten:: dispatch with CPU-side range push/pop, so the query point is the host-side launch timestamp, when cuLaunchKernel returned, not GPU execution time, which would reintroduce the exact async gap the annotation exists to bracket. When multiple ranges enclose one kernel the innermost wins, but the multiplicity is kept: that is the observable signal that Triton fused across operator boundaries.

**Inductor enrichment augments; it never overrides.** In debug mode, Inductor writes each kernel call site under a structured comment naming its constituent operators:

```text
# Original ATen: [aten.relu, aten.addmm]
def triton_poi_fused_relu_addmm_0(in_ptr0, ...):

```

Parsing those yields an exact fusion map, applied as a post-attribution pass with two jobs: promote surviving UNATTRIBUTED kernels to MEDIUM, and populate fused_with on already-attributed kernels without touching their tier. On compiled workloads it typically adds 5–15 percentage points of coverage.

**UNATTRIBUTED is a reported tier, not a dropped kernel.** The three paths correspond to three ways PyTorch hands work to the GPU with graph structure intact. A kernel matching none of them went through a path that preserves no graph structure at all, typically a vendor library entry point like cuDNN's RNN. So a high unattributed fraction isn't a coverage failure to apologize for. It's a measurement, and what it measures is that **your optimization layer is aimed at the wrong altitude.**

## Step 3: Lock the clocks, and never average a counter

A boost clock floats 5–15% with die temperature and power draw, and the bias is directional: an optimized workload burns less power, so it sustains a higher clock and inflates its own speedup. Probe-and-lock pins SM and memory clocks for the nsys capture, a 1.5 s synthetic load after a 1 s warmup, median observed clock as the target, cached per GPU, so baseline and optimized run at identical frequency and the ratio is clock-immune by construction. Standard practice, one paragraph; skipping it quietly manufactures speedup.

The counter set is cut from 90+ to 20 under three constraints: coverage of every bottleneck axis, availability across Ampere, Hopper and Blackwell, and non-redundancy. All renamed variants go in one --metrics request and the first non-null wins. warp_cycles_per_instruction has no Blackwell equivalent with a matching denominator, so it reports null rather than a differently-denominated substitute, the same principle as the unattributed tier. **An honest gap beats a plausible wrong number.**

Aggregation is per-counter, because averaging lies: a 2 μs auxiliary kernel at 90% occupancy and a 200 μs compute kernel at 10% average to a comfortable 50%, and the bottleneck disappears from the summary. Rates are duration-weighted (Σᵢ mᵢ·dᵢ / Σᵢ dᵢ), additive quantities are summed, and per-kernel constants like registers per thread take the maximum. What comes out is one record per operator, provenance attached:

```text
"operator_name": "aten::mm",  "call_index": 26,  "is_fused": false,
"kernels": [{ "kernel_name": "cutlass_80_simt_sgemm_128x32_8x5_nn_align1",
              "attribution_method": "nvtx", "confidence": "medium",
              "duration_ns": 32736,  "grid_dim": [128, 1, 5] }],
"aggregated": { "tensor_core_active_pct": 0.0,   "achieved_occupancy": 18.26,
                "sm_throughput_pct": 37.6,       "dram_throughput_pct": 7.16,
                "l2_hit_rate": 89.55,            "registers_per_thread": 80.0,
                "warp_cycles_per_instruction": null }

```

Tensor Cores idle, occupancy 18%, DRAM at 7% of peak, L2 hit rate 89.6%: not memory-bound, not occupancy-saturated, working set resident in cache, an FP32 GEMM on the SIMT path. And the record says how it knows: NVTX enclosure, medium confidence. One read, and not derivable from any single tool in the table above.

Replay is not free, but its cost stays bounded two ways: application-mode replay re-runs the workload once per counter group rather than once per unique kernel name, O(counter groups) instead of O(unique kernels), a 10–50× reduction on a model like GPT-2, and layer deduplication profiles one representative per structurally identical subgraph, propagating its counters to the duplicates, 12× on GPT-2's twelve identical blocks.

## Three workloads, and what the profile found in each

To test whether attributed profiles carry enough signal to both identify a bottleneck and verify its elimination, we used them to drive an FX graph optimization workflow across three workloads on a single RTX PRO 6000 Blackwell. The optimization layer is a validation vehicle, not the contribution, the same output feeds CI regression detection, hardware-aware architecture search, and kernel-level profiling of serving stacks like vLLM or SGLang.

![](https://cdn.sanity.io/images/wy75wyma/production/62fc9112b0196e2474752ed52ee5d9741ae933c2-1116x433.png)

<!-- unsupported block: table -->

Profiled forward-pass kernel wall time, nsys CUPTI durations at locked clocks. Point estimates from 2 measured iterations. The LSTM baseline is non-representative, nn.LSTM triggers a Dynamo graph break, and its case is diagnostic rather than an FX optimization; restricted to attributed records it is 3.63×.

The totals are the least interesting part. Four things the attributed profile bought that timing alone could not:

**1. It turns a whole-GPU reading into a per-operator budget.** A workload-level ncu report on GPT-2 says tensor_core_active_pct = 0%. True, and nearly useless, it names no operator family and implies no bound on the available gain. The profile resolves it to **87.5% of attributed runtime, 6,401 of 7,313 μs, in the GEMM family**, at 0% Tensor Core activity and 210 registers per thread. The target and its ceiling are both computable before any backend code is written. BF16 promotion rerouted all 96 GEMM kernels off the FP32 SIMT path, and the counters below confirm the mechanism, not just the outcome.

**2. The counter finds the problem; the graph says whether you can fix it.** An aggregate occupancy reading tells you the number is low. It can't tell you whose, and a low number alone implies no particular transformation. On SDPA Attention the profile localizes **occupancy = 16.6% to the Q/K/V projection GEMMs**, holding 96.7% of attributed runtime. The other half of the diagnosis comes from the FX graph: all three projections read from one shared LayerNorm activation, the structural precondition that makes fusion legal. Fusing them into a single [512, 1536] GEMM turned three serial 128-block launches into one 384-block launch and took occupancy to **73.9%**. The chain is per-operator counter → graph-structural check → specific pass, and no link in it comes from aggregate profiling.

**3. The biggest speedup came from an optimization that never ran.** A kernel-level timer on the LSTM encoder shows 1,280 small-matrix GEMMs (M=32, Tensor Cores idle, occupancy 9.4%) with 1,280 matching cublasLt::splitKreduce epilogues. Every instinct that trains points the same way: tile them, promote to BF16, fuse the epilogues. All three are wrong, and nothing in the timing data says so.

The tier does. **88% UNATTRIBUTED through a cuDNN-opaque dispatch path** says the LSTM body is unreachable by any FX-level transformation, because there is no graph to rewrite: nn.LSTM triggers a hard Dynamo graph break via torch._VF.lstm, after which Inductor unrolls it per timestep. The tiny GEMMs are a symptom of the graph break, not a tuning opportunity. The fix is structural, route the recurrent region to cuDNN's fused Tensor-Core RNN, which batches the input-to-hidden projection across all 128 timesteps into one launch (M=32 → M=4096), eliminating every epilogue and 2.26 ms of attributed time.

**Every registered FX pass reported NOT APPLIED**, because the backend callback is never invoked for an eagerly-dispatched region. It is still the largest speedup in the suite. A framework built to apply graph transformations produced its best result by correctly concluding that no graph transformation applied.

<!-- unsupported block: table -->

Duration-weighted counter aggregates per operator class. Every bottleneck the profile named moved, and moved in the direction predicted, the claim a latency number alone cannot support.

**4. It also shows what the optimization cost, not just what it saved.**

![](https://cdn.sanity.io/images/wy75wyma/production/3cd75c7c8b2ebbc87329a04758988e095972b09d-1116x451.png)

The bottom row of that chart is the point. Knowing an optimization netted 1.76× tells you it worked. Knowing it bought 3,706 μs on GEMMs and gave back 659 μs to cast overhead tells you whether a better cast-cancellation pass is worth an afternoon. Only the second is actionable. Same on SDPA, where new auxiliary kernels take 9.1% of the optimized profile, which is precisely what bounds that result at 2.24× rather than higher.

## From a counter reading to a specific fix

FlashAttention, xFormers, fused RNN paths, FP8 promotion, the fix library is mature. What has been missing is the evidence layer that says which one a given operator warrants, instead of applying them by reputation. That is what this table is: the measured symptom on the left, the bottleneck it implies in the middle, the transformation that bottleneck licenses on the right.

<!-- unsupported block: table -->

Rows 1, 2, 3 and 6 are demonstrated end-to-end above; rows 4 and 5 follow from the same bottleneck-axis reasoning but were not exercised in this evaluation.

**Three rules of thumb from the data**

**1. Timing says where; only counters say why.** Three bottlenecks produce the same 6.4 ms and imply three unrelated fixes. Choosing between them without per-operator counters is guessing with extra steps.

**2. Never join on a clock you don't own.** Two tools, two clock domains, accumulating drift, and a join that fails silently, worst exactly where operators are hardest to tell apart. Execution order is a key you control.

**3. What didn't map is a result.** A high unattributed fraction says the work left the graph and no FX-level cleverness will reach it. The suite's largest speedup came from reading that number correctly.

## How this was measured, and what it doesn't cover

Single NVIDIA RTX PRO 6000 Blackwell, locked clocks, 2 warmup + 2 measured iterations, baseline torch.compile(backend="inductor") at FP32 defaults. Durations are nsys CUPTI kernel times; ncu replay supplies counters only and its timings are excluded from every speedup figure.

```text
# requirements
NVIDIA GPU (Ampere minimum) · nsys ≥ 2024.6 · ncu ≥ 2025.4.1
PyTorch 2.11+ · CUDA 12.8

# reproduction scripts, profile.json baseline/optimized pairs,
# and per-example counter comparisons
github.com/yottalabsai/Profiler

```

And the limits, plainly. **The evaluation is narrow:** three workloads, one architecture, no multi-GPU; Ampere and Hopper counter fallbacks are implemented but unvalidated at scale. **Speedups are point estimates** from two measured iterations. **Coverage genuinely degrades on cuDNN-backed operators**, expect 80–90% unattributed for nn.LSTM and nn.MultiheadAttention. The tier makes that legible rather than hidden, but legible isn't solved. **And the pipeline surfaces evidence; it doesn't prescribe actions**, every transformation above was chosen by a human reading a counter.

## Three failures that produce no error message

Each of these returns empty output that looks exactly like a successful run. They shaped the architecture more than any design document did.

**1. torch.profiler and nsys cannot share a process.** nsys's CUPTI subscriber preempts the torch.profiler callback silently: the correlation pass exits normally and writes a .corr.json with zero entries, no warning, no non-zero exit. Hence the two-phase capture, and hence the determinism invariant extending across phases, warmup and measured iteration counts must match in Phase 1, Phase 2 and the ncu replay, or every invocation index shifts.

**2. ncu --filter-by-nvtx-range doesn't work on PyTorch 2.x.** It's the obvious mechanism, confine each replay to kernels inside a named aten:: range. But ncu's injection targets the dynamically-loaded NVTX library, while PyTorch 2.x links NVTX statically inside libtorch_cuda.so. The annotations are invisible to ncu no matter how much emit_nvtx instrumentation you add, and the counters come back empty with no diagnostic.

**3. Per-kernel-name replay doesn't scale.** The remaining obvious option, invoke ncu once per unique kernel name under a name filter, costs O(unique kernels), and GPT-2 alone emits hundreds of distinct Triton variants.

Eliminating all three is what forces application-mode replay: re-execute the workload once per counter group, 4–8 passes, all 20 metrics at once. Not the elegant choice, the only one that reliably collects counters for Inductor-compiled workloads.

The pieces were all on the table: CUPTI can correlate kernels to operators, ncu can measure anything the silicon exposes, nsys can hold the timeline. What was missing was a profile that says where each of its numbers came from, joined on a key both tools agree on, stamped with the evidence behind every attribution, honest about the rows that matched nothing, and measured under conditions held still.

Do that and the profiler stops reporting what is slow and starts reporting which resource it is waiting on. It also becomes a regression signal a CI job can read, a training set of (symptom, transformation, outcome) tuples, and, on occasion, the thing that tells you you are optimizing the wrong layer of the stack entirely.

*Operator Profiler — Logan Chu (Duke University), Dong Li (UC Merced). Full methodology, the 20-counter reference and the annotated profile.json schema are in the technical report.*