This document is relevant for: Trn2, Trn3

How to set up vLLM Neuron#

Overview#

This topic discusses how to install and configure the vLLM Neuron plugin on AWS Trainium or Inferentia so you can serve large language models with vLLM on Neuron. When you have completed it, you will have a working environment ready for the online or offline serving quickstarts.

Note

This guide is for the vLLM Neuron plugin introduced in Neuron 2.31. If you are using vLLM with the NxD Inference library, see the vLLM + NxD Inference documentation. For migration guidance from NxD Inference to vLLM Neuron, see migrate from NxD Inference to vLLM Neuron.

Prerequisites#

  • Instance: A supported Trainium EC2 instance (for example, trn2.48xlarge).

  • Neuron SDK version: Neuron 2.31 or later.

  • Python: Python 3.11 or later.

  • Hugging Face access (optional): An accepted model license and a Hugging Face token if you plan to pull gated models.

Instructions#

Option A: Install from source#

Clone the repository and install the plugin from source:

git clone https://github.com/vllm-project/vllm-neuron.git
cd vllm-neuron
pip install --extra-index-url=https://pip.repos.neuron.amazonaws.com -e .

This installs the vLLM Neuron plugin along with vLLM. The Neuron framework and compiler packages (neuronxcc, torch_neuronx, nki) are not included and must be installed separately (e.g., via the Neuron DLAMI in Option B, or by following the Neuron installation guide).

Option B: Use the Neuron DLAMI#

Launch an instance with the Neuron Multi-Framework Deep Learning AMI. It ships with the Neuron SDK, PyTorch NeuronX, and a pre-built virtual environment that includes vLLM Neuron.

Launch an instance with the DLAMI by following the Neuron DLAMI setup instructions. After you connect, activate the vLLM virtual environment:

source /opt/aws_neuronx_venv_pytorch_inference_vllm_0_21_0_1_0_0/bin/activate

Option C: Use a container#

Use the vLLM Inference NeuronX DLC which bundles the SDK and all dependencies.

Verify the installation#

Confirm that vLLM discovers the Neuron platform plugin:

python -c "import vllm; from vllm.platforms import current_platform; print(current_platform.device_name)"

The output should be:

neuron

Confirm the Neuron runtime is visible:

neuron-ls

You should see output listing the available NeuronCores on your instance.

Install dependencies for disaggregated inference#

Note

This section is only required if you plan to run disaggregated inference (DI), which requires the NIXL LIBFABRIC transport ("backends": ["LIBFABRIC"]) for KV cache transfer over EFA. If you are not using it, skip ahead to Configure environment variables.

To use the NIXL LIBFABRIC backend for disaggregated inference, the following two shared libraries must be installed:

  • libcuda.so.1 — the CUDA driver library. The plugin links against it for its optional GPUDirect path, which is never exercised on Neuron, so a stub is sufficient.

  • libfabric.so.1 — the Libfabric/EFA library that carries the actual RDMA transfer.

The backend ships as a plugin (libplugin_LIBFABRIC.so) inside the nixl wheel that NIXL dlopens when you start a server with "backends": ["LIBFABRIC"]. If either library is missing from the loader path, dlopen fails and NIXL reports createBackend: unsupported backend 'LIBFABRIC' followed by nixlNotFoundError: NIXL_ERR_NOT_FOUND, and every server worker exits.

Some environments already ship one or both of these libraries. Run the checks below and provision only the libraries that are actually missing.

1. Check for libcuda.so.1#

ldconfig -p | grep libcuda.so

If this prints a libcuda.so.1 line, the CUDA stub is already present — skip to step 2. If it prints nothing, install the official CUDA driver stub (as root) and expose it as libcuda.so.1:

# Add the NVIDIA CUDA apt repo (the ubuntu2204 keyring works on Ubuntu 24.04)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
dpkg -i cuda-keyring_1.0-1_all.deb && rm -f cuda-keyring_1.0-1_all.deb
apt-get update

# Minimal package that ships the CUDA driver stub (the full toolkit is not needed)
apt-get install -y --no-install-recommends cuda-cudart-dev-12-6

# Expose the stub as libcuda.so.1 on the system loader path.
# /usr/local/lib is already on the default ldconfig search path, so the
# symlink plus ldconfig is enough — no ld.so.conf.d entry is needed.
ln -sf /usr/local/cuda-12.6/targets/x86_64-linux/lib/stubs/libcuda.so /usr/local/lib/libcuda.so.1
ldconfig

If you cannot install packages (for example, a read-only or network-isolated container), put the directory containing a libcuda.so.1 stub on LD_LIBRARY_PATH in the same shell that runs vllm serve instead of using ldconfig:

export LD_LIBRARY_PATH=/path/to/dir/with/libcuda.so.1:$LD_LIBRARY_PATH

Note

A CUDA driver is a stub library (Error 34) warning during server startup is expected and harmless — nothing on the Neuron/EFA path calls into the CUDA driver.

2. Check for libfabric.so.1 (EFA)#

ldconfig -p | grep libfabric.so
fi_info -p efa   # should list at least one EFA provider

If libfabric.so.1 resolves and fi_info -p efa lists a provider, EFA is ready — skip to the verification step. If libfabric.so.1 is not found, install the AWS EFA installer (userspace libraries only — the host supplies the kernel driver):

cd /tmp
curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz
tar -xf aws-efa-installer-latest.tar.gz
cd aws-efa-installer
# --skip-kmod installs userspace libraries only (the host supplies the EFA
# kernel driver); --no-verify skips the installer's kernel-module verification
# step, which otherwise reports a false failure inside a container even though
# --skip-kmod is set.
bash efa_installer.sh --yes --skip-kmod --no-verify

This installs Libfabric under /opt/amazon/efa. Add its library directories to LD_LIBRARY_PATH in the same shell that runs vllm serve so the loader can find libfabric.so.1:

export LD_LIBRARY_PATH=/opt/amazon/efa/lib64:/opt/amazon/efa/lib:$LD_LIBRARY_PATH
export PATH=/opt/amazon/efa/bin:$PATH   # for fi_info and other EFA tools

If Libfabric is installed but simply not on the loader path (for example, fi_info -p efa works but ldconfig -p | grep libfabric.so is empty), setting LD_LIBRARY_PATH as above is all that is needed.

3. Verify the LIBFABRIC backend loads#

Before launching the server, confirm the backend instantiates end-to-end:

python -c "from nixl._api import nixl_agent, nixl_agent_config; \
nixl_agent('probe', nixl_agent_config(backends=['LIBFABRIC'])); print('LIBFABRIC OK')"

A successful run prints NIXL INFO Backend LIBFABRIC was instantiated followed by LIBFABRIC OK. If you still see unsupported backend 'LIBFABRIC', run ldd on the plugin to see which library is still unresolved:

ldd "$(find "$(python -c 'import site; print(site.getsitepackages()[0])')" \
    -name libplugin_LIBFABRIC.so | head -1)" | grep 'not found'

Resolve any not found entry by adding its directory to LD_LIBRARY_PATH. Entries such as libcudart.so.13 are used only by the GPUDirect path and can be left unresolved on Neuron.

Configure environment variables#

The following environment variables control common behaviors:

Variable

Purpose

Default

VLLM_CACHE_ROOT

Root directory for the Neuron compile cache. Set to local NVMe for best performance.

~/.cache/vllm

VLLM_NEURON_LOG_LEVEL

Plugin log verbosity (DEBUG, INFO, WARNING, ERROR).

INFO

HF_TOKEN

Hugging Face access token for gated model downloads.

(none)

 
export VLLM_CACHE_ROOT=/local/cache/vllm
export HF_TOKEN=hf_your_token_here

Confirm your work#

Run a minimal vllm serve command to confirm the full stack is wired up. This example uses gpt-oss-20b with a small context and a single compiled bucket to keep compilation time short:

vllm serve openai/gpt-oss-20b \
    --tensor-parallel-size 8 \
    --max-num-seqs 1 \
    --max-model-len 1024 \
    --max-num-batched-tokens 512 \
    --hf-overrides '{"quantization_config": {}}' \
    --additional-config '{"neuron_config": {"num_batched_tokens_buckets": [512], "num_seqs_buckets": [1]}}'

Wait for the server to print Uvicorn running on .... Then, in a second terminal on the same instance:

curl -s http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-oss-20b",
      "messages": [{"role": "user", "content": "Hello, Neuron!"}],
      "max_tokens": 16
    }'

A JSON response containing generated text confirms that the plugin, Neuron runtime, and model compilation are all working. Stop the server with Ctrl+C.

Next steps#

Common issues#

Plugin fails to load the Neuron platform#

  • Possible solution: Confirm you activated the correct virtual environment and that pip show vllm-neuron reports version 0.21.0.1.0.0 or later. If you installed vLLM separately before the plugin, reinstall the plugin to ensure version alignment.

neuron-ls shows no devices#

  • Possible solution: Verify you are on a Neuron-supported instance type (trn2 or trn3). On a fresh instance, the Neuron runtime may need a reboot after driver installation. Check dmesg | grep neuron for driver messages.

Compilation times out or is very slow on first run#

  • Possible solution: First-time model compilation on Neuron can take several minutes depending on model size. Subsequent runs use the compile cache. Ensure VLLM_CACHE_ROOT points to fast local storage (NVMe) rather than a network filesystem.

Python version mismatch#

  • Possible solution: The vLLM Neuron plugin requires Python 3.11 or later. Run python --version to check. The DLAMI virtual environment ships with a compatible Python version.