This document is relevant for: Trn2, Trn3

vLLM integration

Design documentation for how vLLM Neuron integrates with the vLLM core framework. For user-facing configuration, see the features guide and configuration options.

Topic	Description
vLLM integration design reference	Plugin registration, scheduler, KV cache
KV cache integration	KV cache integration points with vLLM
Additional config	Additional configuration options
Async scheduling and execution	Async scheduling and execution design
Context-length bucketing	Reducing decode HBM reads
Memory management	Available memory determination
KV cache quantization	FP8 KV cache mechanics
Metrics	Production metrics design
Neuron profiling	Profiling integration
Neuron scheduler	Holdback queue and admission control
Block KV batching and padding	Block KV cache architecture
Prefix caching	Prefill segmentation and KV reuse
Pooling models	Pooling model execution on Neuron
Prompt embeddings	Prompt embedding support
Structured outputs and tool calling	JSON, regex, grammar-constrained generation
Disaggregated inference	DI architecture, NIXL transport, hybrid TP, DCP

This document is relevant for: Trn2, Trn3