This document is relevant for: Trn2, Trn3

vLLM Neuron Plugin (Beta) Documentation#

The vLLM Neuron plugin brings the full vLLM serving stack to AWS Trainium accelerators. It supports continuous batching, speculative decoding (EAGLE3), disaggregated inference, structured outputs, multimodal models, and more — all accessible through the standard vllm serve command and OpenAI-compatible API.

For a high-level overview of inference on Neuron and help choosing the right inference solution, see Inference on Neuron. The source code for the vLLM Neuron plugin is hosted in the vLLM Neuron GitHub repository.

Get started#

Setup guide

Install and configure vLLM Neuron on a Trainium instance.

Online serving quickstart

Launch an OpenAI-compatible API server and send your first chat request.

Offline serving quickstart

High-throughput batch inference with the vllm.LLM Python API.

Migration from NxD Inference

Migrate existing NxDI deployments to the vLLM Neuron plugin.

Deploy & serve#

Features guide

Configure and tune all serving features — bucketing, quantization, DI, speculation, and more.

Configuration reference

All Neuron-specific options in additional_config and environment variables.

Profiling workloads

Capture Neuron Runtime profiles via built-in profiler endpoints.

Models#

Deploy GPT-OSS

Model recipe for GPT-OSS 20B and 120B (MoE) on Trn2/Trn3.

Tutorials#

Disaggregated inference: 1P1D and xPyD

Configure disaggregated inference topologies.

Prefix caching benchmark

Measure TTFT improvement from prefix caching with GPT-OSS.

Model development#

Onboard a new model

Implement and register a new architecture with vLLM.

CPU development workflow

Develop and test without Neuron hardware.

NKI CPU simulator

Validate NKI kernel correctness on CPU.

Debugging accuracy issues

Methodology for isolating where accuracy drift is introduced.

Concepts & architecture#

Parallelism#

Topic	Description
Data parallelism	Data parallelism overview
Expert parallelism	Expert parallelism for MoE
Tensor parallelism	Tensor parallelism overview

Multimodal#

Topic	Description
M-RoPE	Spatial position embeddings for VLMs

vLLM integration#

Topic	Description
KV cache integration	KV cache integration points with vLLM
Async scheduling and execution	Async scheduling and execution design
Metrics	Production metrics design
Neuron profiling	Profiling integration
Neuron scheduler	Holdback queue and admission control
Prefix caching	Prefill segmentation and KV reuse
Disaggregated inference	DI architecture, NIXL transport, hybrid TP