This document is relevant for: Trn2, Trn3

Tutorial: Deploy Qwen3-VL-32B with vLLM Neuron#

This tutorial walks through deploying Qwen3-VL-32B-Instruct on a trn2.48xlarge instance using vLLM-Neuron. It covers environment setup, model download, online serving, and offline inference.

Prerequisites:

A trn2.48xlarge instance (16 NeuronCores) with Neuron SDK 2.31.0 or later. See setup guide.
vLLM Neuron plugin 0.21.0 or above installed.
Python 3.10+

Step 1: Environment Setup#

Verify Neuron devices are visible:

neuron-ls
# Expected: 16 NeuronCores listed for trn2.48xlarge

Set environment variables before running any inference script:

# Extend timeouts for large model compilation
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200
export VLLM_NEURON_COMPILATION_TIMEOUT=1200

# Required if your home directory is on NFS
export NEURON_CC_FLAGS="--temp-dir=/tmp/neuroncc_tmp"
mkdir -p /tmp/neuroncc_tmp

Step 2: Download the Model#

huggingface-cli download \
    Qwen/Qwen3-VL-32B-Instruct \
    --local-dir /path/to/Qwen3-VL-32B-Instruct

Tip: On a trn2 cluster, download to a shared filesystem instead of your home directory to avoid NFS write issues.

Step 3: Online Serving#

Start a vLLM OpenAI-compatible server:

vllm serve /path/to/Qwen3-VL-32B-Instruct \
    --served-model-name Qwen3-VL-32B-Instruct \
    --max-model-len 32768 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 8 \
    --tensor-parallel-size 16 \
    --additional-config '{
        "neuron_config": {
            "quantization": "bf16",
            "num_batched_tokens_buckets": [4096],
            "num_seqs_buckets": [8],
            "on_device_sampling_config": {"all_greedy": true}
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048],
            "vision_attention_block_size": 2048
        }
    }'

Once the server is up, send requests using the OpenAI Python SDK:

import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

# Text-only
response = client.chat.completions.create(
    model="Qwen3-VL-32B-Instruct",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=50,
)
print(response.choices[0].message.content)

# Image + text
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="Qwen3-VL-32B-Instruct",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
        {"type": "text", "text": "Describe this image."},
    ]}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Step 4: Offline Inference#

vLLM-Neuron compiles the model on the first run and caches the artifacts to ~/.cache/vllm/neuron/compile_cache. Subsequent runs skip recompilation and load from cache.

Configuration#

The model has two components, each with its own config object passed via additional_config (see the configuration options reference):

neuron_config: text decoder settings (token/sequence bucket sizes, sampling, quantization).
vision_neuron_config: vision encoder settings (vision token buckets, attention block size, and optional vision TP/DP split).

Bucket sizes control the discrete padded shapes compiled into each NEFF. Each bucket adds compile time; start with one and add more as needed. The two components bucket along different dimensions, so their buckets are configured separately:

num_batched_tokens_buckets (in neuron_config): text-decoder buckets over the number of batched text tokens per forward pass. See compilation options.
num_vision_tokens_buckets (in vision_neuron_config): vision-encoder buckets over the number of vision patches per encoder forward pass (raw T*H*W patches from image_grid_thw, before the 2x2 spatial merge; this is the count select_vision_bucket matches against). The scheduler may batch images from multiple requests into one forward pass, so size the buckets for the total images processed together, not a single request. These scale with image count and resolution:

`num_vision_tokens_buckets`	Approximate capacity
`[2048]`	1–2 images at 448×448 px
`[2048, 8192]`	Up to ~8 images
`[2048, 8192, 20480]`	Up to ~20 images

Run offline inference#

The following script runs text-only, single-image, and multi-image inference:

import os

os.environ["VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS"] = "1200"
os.environ["VLLM_NEURON_COMPILATION_TIMEOUT"] = "1200"

from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset

MODEL_PATH = "/path/to/Qwen3-VL-32B-Instruct"

llm = LLM(
    model=MODEL_PATH,
    max_model_len=32768,
    max_num_batched_tokens=4096,
    max_num_seqs=8,
    tensor_parallel_size=16,
    additional_config={
        "neuron_config": {
            "quantization": "bf16",
            "num_batched_tokens_buckets": [4096],
            "num_seqs_buckets": [8],
            "on_device_sampling_config": {"all_greedy": True},
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048],
            "vision_attention_block_size": 2048,
        },
    },
)

processor = AutoProcessor.from_pretrained(MODEL_PATH)
sampling_params = SamplingParams(max_tokens=200, temperature=0.0)

# --- Text-only ---
outputs = llm.generate(["What is the capital of France?"], sampling_params)
print(outputs[0].outputs[0].text)

# --- Single image ---
image = ImageAsset("cherry_blossom").pil_image.resize((640, 320))
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate([{"prompt": prompt, "multi_modal_data": {"image": [image]}}], sampling_params)
print(outputs[0].outputs[0].text)

# --- Multi-image ---
images = [
    ImageAsset("stop_sign").pil_image.resize((448, 448)),
    ImageAsset("cherry_blossom").pil_image.resize((448, 448)),
]
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "image"},
    {"type": "text", "text": "Compare these two images."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate([{"prompt": prompt, "multi_modal_data": {"image": images}}], sampling_params)
print(outputs[0].outputs[0].text)

Vision Parallelism (Optional)#

By default the vision encoder runs as 16 independent DP replicas (TP1 DP16), one per NeuronCore, the recommended layout for high-throughput multi-image workloads. To shard the encoder weights across cores for a single large image instead, increase tp_size.

Set tp_size inside vision_neuron_config to change the split. DP is derived automatically as world_size / tp_size:

`tp_size`	Vision TP	Vision DP	Best for
`1` (default)	1	16	Multi-image, high throughput
`4`	4	4	Mixed workloads
`16`	16	1	Single-image, low latency

Example — TP1 DP16:

"vision_neuron_config": {
    "num_vision_tokens_buckets": [2048],
    "vision_attention_block_size": 2048,
    "tp_size": 1,  # DP = world_size / tp_size = 16
}

Conclusion#

You have successfully deployed Qwen3-VL-32B-Instruct on a trn2.48xlarge instance. The model inference on Trainium currently supports text-only, single-image, multi-image, and video inputs via both the offline LLM API and the OpenAI-compatible online serving endpoint. For accuracy validation results, see the model card.