vLLM Neuron: LLM serving on Neuron#

Note

vLLM Neuron is currently in Beta.

The vLLM Neuron plugin brings the full vLLM serving stack to AWS Trainium (Trn2 and Trn3). It is the recommended path for serving large language models (LLMs) on Neuron, exposing the standard vllm serve command and an OpenAI-compatible API. This page is the section home for vLLM Neuron: it points you to the setup guide, quickstarts, feature and configuration references, model recipes, and tutorials.

The plugin supports continuous batching, prefix caching, speculative decoding (EAGLE3), disaggregated inference, structured outputs, quantization, and multimodal models. Its documentation is sourced from the vLLM Neuron plugin repository and maintained alongside the plugin itself.

Applies to#

This section is applicable to:

  • Online, OpenAI-compatible LLM serving on AWS Trainium (Trn2/Trn3).

  • High-throughput offline batch inference with the vllm.LLM Python API.

  • Migrating existing NxD Inference deployments to the vLLM Neuron plugin.

Get started#

Setup guide

Install and configure vLLM Neuron on a Trainium instance.

Online serving quickstart

Launch an OpenAI-compatible API server and send your first chat request.

Offline serving quickstart

Run high-throughput batch inference with the vllm.LLM Python API.

Migrate from NxD Inference

Move an existing NxDI deployment to the vLLM Neuron plugin.

Deploy and serve#

Features guide

Configure and tune serving features — bucketing, quantization, disaggregated inference, and speculation.

Configuration reference

All Neuron-specific options in additional_config and the supported environment variables.

Model recipes

Supported models and their feature tables — Llama 3, GPT-OSS, Qwen3-VL, and more.

Tutorials

End-to-end walk-throughs for speculative decoding, disaggregated inference, and multimodal serving.

Develop and understand#

Model development

Onboard new architectures, add vision-language models, and debug accuracy on Neuron.

Concepts and architecture

Design deep dives — parallelism, speculation, compilation, and vLLM integration internals.

The source code for the vLLM Neuron plugin is hosted in the public vLLM Neuron GitHub repository, where external contributors can clone the plugin and open documentation pull requests.