This document is relevant for: Trn2, Trn3

Concepts & architecture

How vLLM Neuron works under the hood — parallelism strategies, plugin integration, speculative decoding internals, and the accuracy validation framework.

Parallelism

Topic	Description
Attention DP	Sharding Q/O weights across DP groups
Component DP sharding	Per-component independent sharding
Data parallelism	Data parallelism overview
Decode Context Parallelism	KV cache sequence sharding for long contexts
Expert parallelism	Expert parallelism for MoE
Tensor parallelism	Tensor parallelism overview
Vision encoder parallelism	Independent TP/DP for vision encoders

Speculation

Topic	Description
Speculative decoding	EAGLE3 internals on Neuron

Multimodal

Topic	Description
Block Packing Attention	FFD block packing for multi-image attention efficiency
On-Device Encoder Cache	Block-based on-device cache for vision encoder outputs
M-RoPE	Spatial position embeddings for VLMs

vLLM integration

Topic	Description
vLLM integration design reference	Plugin registration, scheduler, KV cache
KV cache integration	KV cache integration points with vLLM
Additional config	Additional configuration options
Async scheduling and execution	Async scheduling and execution design
Context-length bucketing	Reducing decode HBM reads
Memory management	Available memory determination
KV cache quantization	FP8 KV cache mechanics
Metrics	Production metrics design
Neuron profiling	Profiling integration
Neuron scheduler	Holdback queue and admission control
Block KV batching and padding	Block KV cache architecture
Prefix caching	Prefill segmentation and KV reuse
Pooling models	Pooling model execution on Neuron
Prompt embeddings	Prompt embedding support
Structured outputs and tool calling	JSON, regex, grammar-constrained generation
Disaggregated inference	DI architecture, NIXL transport, hybrid TP, DCP

Framework

Topic	Description
Async execution	Async execution double buffering
Model bringup	Model bringup workflow
Model factory	Model registry and factory pattern

Compilation

Topic	Description
Compilation cache	Compilation cache (hit/miss, remote store)
CPU compilation	Ahead-of-time CPU compilation (NEFF extraction)
FX passes architecture	FX passes architecture
Aliasing output rewrite pass	Aliasing output rewrite pass
Device rewriting FX pass	Device rewriting FX pass
Inplace to out-of-place pass	Inplace to out-of-place rewrite

Accuracy

Topic	Description
Accuracy debugging design	Accuracy debugging framework
KV cache analysis	KV cache analysis
Logit validation	Logit validation
Module test guidelines	Module test guidelines
Tensor capture	Tensor capture
Tensor compare	Tensor compare
Tensor replacement	Tensor replacement

This document is relevant for: Trn2, Trn3