This document is relevant for: Trn2, Trn3

Tutorial: Analyze a vLLM Neuron profile using the Profile Analysis skill

This tutorial walks you through analyzing a profile you have already captured. You point the agent at a capture manifest, tell it what you want to learn, and work through the answers with it: which compiled graphs ran, where a step sits relative to its hardware floor, which engine binds it, how much of it is collective traffic, and which part of the model is responsible.

The agent reads both halves of the capture — the system trace for the session-level picture and the step time, the device trace for the per-instruction detail inside a step.

By the end you will have a per-phase analysis block of machine-readable findings and a written report, and you will know how to check that both are sound rather than merely plausible.

Important

Use this skill interactively, and say what you want to know. An analysis has no single correct output — the useful answer depends on the question. Asking “analyze this profile” produces a general survey; asking “why is decode 2.5 times its memory floor” drives the stages that bear on it. Plan to converse with the agent rather than to hand it a job and come back.

Note

This tutorial starts from a capture. Produce one first with Tutorial: Capture prefill and decode profiles using the Profiling skill. For what the numbers mean and which to trust, see the Profile Analysis deep dive.

Prerequisites

Have a completed capture

You need a run directory with a manifest.json whose device trace is real. This tutorial uses the Gemma 4 31B IT capture from Tutorial: Capture prefill and decode profiles using the Profiling skill — tensor parallel 16, 8192 tokens in and 1024 out.

cd ~/profiling-tutorial/agent_artifacts/profiles
python3 -c "
import json, os
m = json.load(open('gemma4_decode/manifest.json'))
n = m['profile']['device_ntff']
print('recipe:', m['recipe'])
print('trace :', n, os.path.getsize(n) if n else 'MISSING')
print('neffs :', len(m['neff_hashes']))
"

A null device trace, or a size of a few hundred kilobytes, means the capture failed and there is nothing to analyze. Go back to the capture tutorial.

Important

The capture must have recorded at least two iterations. At one, CcOp.parquet comes out empty and the collective stage cannot run. The capture tutorial’s recipes use two by default; if you captured by hand, check profiler_config.max_iterations in run_spec.json before continuing.

No Trainium device needed

Analysis is host-side. You can do it on the instance that captured, or copy the capture elsewhere. What you do need is headroom:

df -h .          # disk
free -g          # host memory

Ingesting a long-sequence prefill trace is the slow, memory-hungry step of this tutorial: it runs for a while, writes parquet several times larger than the trace, and holds a substantial amount of host RAM throughout. Decode is far cheaper. If you are short on either, start with the decode phase. For measured figures, see the deep dive’s requirements.

Install Neuron Agentic Development

If you have not already, follow the Getting Started guide and confirm the deploy step completed.

# For Claude Code
deploy-neuron-agentic-development-to-claude

# For Kiro
deploy-neuron-agentic-development-to-kiro

Activate your Python environment and verify the tools

source ~/opt/aws_neuronx_venv_pytorch_2_9/bin/activate
which neuron-explorer
python3 -c "import pandas, pyarrow; print(pandas.__version__, pyarrow.__version__)"

neuron-explorer ingests the capture; pandas and pyarrow query the result.

Step 1. Point the agent at the capture

The manifest is the entry point. Every path in it is absolute, so the agent needs nothing else to locate the artifacts.

export NEURON_PROJECT_ROOT=~/profiling-tutorial

Input	What it is	Example value
manifest path	The capture to analyze. One per phase.	.../profiles/gemma4_decode/manifest.json
NEURON_PROJECT_ROOT	Where the ingested parquet and the report land, beside the capture.	~/profiling-tutorial
your question	What you want to decide. This shapes the whole analysis.	“Is decode memory-bound or collective-bound?”

Step 2. Invoke the analysis agent

Open your agentic IDE and state the capture and the question.

Analyze the decode profile at
~/profiling-tutorial/agent_artifacts/profiles/gemma4_decode/manifest.json

I want to know whether this step is memory-bound or collective-bound,
and which model component accounts for most of the device time.

The agent ingests, then works through the stages that bear on your question. Expect it to come back to you — to confirm which NEFF to treat as the decode graph, or to ask whether a gap matters for your decision. That is the skill working as intended; see Workflow for what each stage does.

Useful follow-ups once the first pass lands:

Break the step down by collective segment and tell me which segment
holds the binding engine.

Compare the MLP and attention roll-up by HBM bytes read, not just by
engine time.

The measured step is well above the floor. Where is the unexplained
time going?

Step 3. What happens during the analysis

The agent ingests the capture into parquet, reads the device constants from the capture itself, establishes the step time from the live dispatch records, computes the floors, partitions the step at the collectives, and rolls the instructions up by model component. The deep dive’s Workflow explains each stage and why it is ordered that way.

Two things you will see that are worth understanding as they happen:

• The ingest is the slow part, and its cost tracks trace size, not model size.

• A prefill capture shows two NEFFs, because the single request also warms the batch-size-1 decode graph. The agent filters to the prefill NEFF; if it asks which to use, the manifest’s neff_hashes is the answer.

Step 4. Read the analysis block

The machine-readable findings land as a per-phase analysis block. This is the output to check first, because every claim in the written report should be traceable to it.

Field	Meaning
measured_step_us	The step time from the live dispatch records.
memory_floor_us, compute_floor_us	The two roofline floors. The larger one bounds the step.
mbu_percent, mfu_percent	Achieved fraction of peak bandwidth and peak FLOPS.
binding_engine, binding_engine_pct	The engine busiest for the largest share of the step.
engine_active_pct	Per-engine active-wall share, interval-merged.
collectives	One entry per operation, with count, wall time, and bytes.
component_rollup	Device time and HBM bytes per model component.
entitlement	The floor the measured step is compared against, its source, and the ratio.

Here is the real decode block from this tutorial’s capture — Gemma 4 31B IT at tensor parallel 16, 8192 in and 1024 out. Trimmed to the largest roll-up entries; the full block listed fourteen.

{
  "measured_step_us": 208296.158,
  "memory_floor_us": 82611.251,
  "compute_floor_us": 1406.57,
  "mbu_percent": 39.66,
  "mfu_percent": 0.68,
  "binding_engine": "sync",
  "binding_engine_pct": 67.35,
  "engine_active_pct": {
    "tensor": 3.99, "vector": 4.21, "scalar": 2.77,
    "gpsimd": 6.25, "sync": 67.35
  },
  "collectives": [
    {"operation": "ALLREDUCE", "count": 121, "wall_us": 3481.15, "bytes_out": 10407936},
    {"operation": "BARRIER",   "count": 1,   "wall_us": 1172.08, "bytes_out": 0},
    {"operation": "ALLGATHER", "count": 3,   "wall_us": 74.25,   "bytes_out": 8454144}
  ],
  "component_rollup": [
    {"module": "Gemma4Model[model]/Gemma4DecoderLayer/Gemma4Attention[self_attn]",
     "engine_us": 344971.35, "hbm_read_bytes": 56407115950},
    {"module": "Gemma4Model[model]/Gemma4DecoderLayer/Gemma4MLP[mlp]",
     "engine_us": 21250.15, "hbm_read_bytes": 2622412800},
    {"module": "Gemma4Model[model]/Gemma4DecoderLayer",
     "engine_us": 3510.06, "hbm_read_bytes": 86228},
    {"module": "ColumnParallelLinear[lm_head]",
     "engine_us": 1459.37, "hbm_read_bytes": 0}
  ],
  "entitlement": {"source": "self", "predicted_us": 82611.25, "ratio": 2.5214}
}

How to read it

Working through it in the order that answers the question asked in Step 2:

• Memory-bound or compute-bound? The memory floor is 82,611 µs against a compute floor of 1,407 µs — a factor of 59. The utilization figures agree: 39.7% of peak bandwidth against 0.68% of peak FLOPS. This step moves bytes; it barely computes. Decode at a long context is expected to look like this.

• At the floor, or above it? 208,296 µs measured against an 82,611 µs floor is a ratio of 2.52, so roughly 126 ms is going somewhere neither floor explains. That gap is the thing worth chasing.

• Collective-bound? No — and this is where reading only one field would mislead you. The collectives total about 4,728 µs, a little over 2% of the step. The 121 all-reduces are the expected tensor-parallel pattern at this degree, and a useful check that the sharding is what you configured, but they are not where the time is.

• So what binds? The sync engine, at 67.35% of the step, while every compute engine sits below 7%. The step is dominated by synchronization that is not collective traffic — the collectives account for 2% while sync accounts for 67%. That distinction is the finding, and it points the investigation at serialization and waits rather than at kernels.

• Which component? Attention at 344,971 µs against MLP at 21,250 µs, reading 56.4 GB against 2.6 GB. For a memory-bound step the bytes column is the one that decides, and attention reads 21 times the MLP’s. At an 8192-token context that is the KV cache, which is exactly where decode attention time goes.

Two checks worth making on any roll-up, including this one:

• Is there an ``Unknown`` entry, and how large? This block has none, which means module scope was populated for every attributed instruction. A large unknown share means the attribution is incomplete — usually a NEFF compiled without framework debug information. See Component attribution.

• Does it name components or instructions? Here each entry is a module path with the layer index collapsed, so attention across all layers is one row. A list of per-layer entries has not answered the question.

Note

Engine times in a roll-up are per-engine interval sums and will overshoot the step where engines overlap — 344,971 µs of attention inside a 208,296 µs step is not a contradiction. Use them to rank components, not to partition the step.

Important

This capture’s prefill phase is a worked example of a failure, not a success: its ingest completed only 30 of the expected tables, dropping the framework and BIR layers while leaving the instruction and collective tables intact. Five of the six analysis dimensions still scored clean on it. Only the table inventory caught it — which is why check 1 in Confirmation is not a formality.

Step 5. Read the report

The agent also writes a report under analysis/ in the run’s output directory, <phase>_report.md. A sound one runs a few thousand bytes and covers: the bottom line, both floors, per-engine active-wall, the collectives, the component roll-up, the entitlement comparison, and a verification that the phase is what the label claims.

Read it against the block from Step 4. If a sentence in the report has no corresponding field in the block, treat it as unsupported.

Confirmation

An analysis fails quietly — a plausible but wrong floor reads exactly like a right one. Run these five checks before acting on any of it.

1. Ingestion is complete. Around 47 tables for the phase, and the attribution tables non-empty.

NE=$NEURON_PROJECT_ROOT/agent_artifacts/data/gemma4_decode/profiles
find "$NE" -name '*.parquet' | wc -l
find "$NE" -name 'CcOp.parquet' -size -1k   # prints a path only if it is empty

An empty CcOp.parquet means the capture recorded one iteration and the collective findings are absent rather than zero.

2. The measured step is at or above the larger floor. If a floor exceeds the measured step, the floor is wrong — almost always single-core device constants applied to an LNC=2 profile.

3. No engine’s active time exceeds the step. If one does, the times were summed rather than interval-merged — see active-wall time.

4. Summed collective wall time is less than the step. Collectives happen inside the step, so exceeding it means the intervals were summed rather than merged.

5. The roll-up names components, not instructions. Entries should read like .../Gemma4MLP[mlp], one per component with the layer index collapsed. A list of per-layer entries has not answered the attention-versus-MLP question. And check the Unknown share: if it dominates, the attribution is not usable.

Step 6. Act on the result

Rank the components, then optimize the binding one — and confirm it is binding before you start, because a correct optimization aimed at a non-binding operation buys nothing.

To verify a change, capture again and compare against this analysis. Compare numbers from the same capture only — a device time from one run against a wall clock from another is arithmetic, not measurement.

Important

Device time is not end-to-end latency. It excludes queueing, scheduling, detokenization, and the serving stack. When you report a result, say which quantity you measured.

Troubleshooting

The deep dive carries the full reference at Common issues. Three things bite specifically during the steps above.

The ingest runs for hours, or writes zero-byte parquet. The trace is too large. Check profiler_config.max_iterations in the capture’s run_spec.json — it should be 2. A capture made with a large iteration cap is faster to redo than to ingest.

A second ingest will not start. neuron-explorer view binds a fixed port even in ingest-only mode, so prefill and decode cannot be ingested at the same time on one host. Run them in sequence.

The component roll-up is mostly ``Unknown``. The NEFF was compiled without framework debug information, which cannot be added after the fact. Recompile the capture with the three debug variables set — see Configuration — and recapture.

Everything else is in Common issues, which covers empty tables, floors above the measured step, engine times exceeding the step, device time not matching benchmark latency, and the profiler schema defects that cause a misreading.

Related resources

• Deep dive: Localize performance bottlenecks with the Profile Analysis skill for the deep dive: the stages, what the numbers mean, the verification rules, and which outputs to trust.

• Tutorial: Capture prefill and decode profiles using the Profiling skill for producing the capture this tutorial consumes.

• Deep dive: Capture device and system profiles from a vLLM Neuron server with the Profiling skill for the capture deep dive.

• Profile Parquet Schema Reference for the authoritative table and field reference.

• Getting Started for installation and setup.

• Neuron Agentic Development GitHub for source code and issue tracker.

This document is relevant for: Trn2, Trn3