This document is relevant for: Trn2, Trn3

Concepts & architecture

How vLLM Neuron works under the hood — parallelism strategies, plugin integration, speculative decoding internals, and the accuracy validation framework.

Parallelism

Topic	Description
Data parallelism	Data parallelism overview
Expert parallelism	Expert parallelism for MoE
Tensor parallelism	Tensor parallelism overview
Vision encoder parallelism	Independent TP/DP for vision encoders

Multimodal

Topic	Description
Block Packing Vision Attention	FFD block packing for multi-image attention efficiency
On-Device Encoder Cache	Block-based on-device cache for vision encoder outputs
M-RoPE	Spatial position embeddings for VLMs

vLLM integration

Topic	Description
KV cache integration	KV cache integration points with vLLM
Async scheduling and execution	Async scheduling and execution design
Metrics	Production metrics design
Neuron profiling	Profiling integration
Neuron scheduler	Holdback queue and admission control
Prefix caching	Prefill segmentation and KV reuse
Disaggregated inference	DI architecture, NIXL transport, hybrid TP

This document is relevant for: Trn2, Trn3