This document is relevant for: Trn2, Trn3

Tutorial: Deploy Qwen3-VL-32B with vLLM Neuron#

This tutorial walks through deploying Qwen3-VL-32B-Instruct with vLLM Neuron. It covers environment setup, model download, an optional MXFP8 quantization step, online serving, and offline inference.

The model runs in two precisions:

BF16 — the default, on trn2.48xlarge or Trn3.
MXFP8 — micro-scaled FP8 weights for the text transformer, Trn3-only. Reduces the weight footprint and improves both throughput and latency while keeping accuracy close to BF16.

Precision	Instance	Steps
BF16	Trn2 or Trn3	1 → 2 → 4 (skip the MXFP8-only Step 3)
MXFP8	Trn3 only	1 → 2 → 3 → 4

BF16 is the main path throughout; each step calls out the MXFP8 differences where they apply. Step 3 (quantization) is only for MXFP8 — skip it for BF16.

Prerequisites:

A trn2.48xlarge instance with Neuron SDK 2.31.0 or later. See setup guide. MXFP8 requires a Trn3 instance (the native MXFP8 kernels are Trn3-only) with Neuron SDK 2.32 or later.
vLLM Neuron plugin 0.21.0 or above installed.
Python 3.10+
For MXFP8, LLM-Compressor to quantize the checkpoint: pip install llmcompressor compressed-tensors

Step 1: Set up your environment#

Verify Neuron devices are visible:

neuron-ls
# Lists the Neuron devices and cores available on your instance.

Set environment variables before running any inference script:

# Extend timeouts for large model compilation
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200
export NEURON_LIBTORCH_COMPILATION_TIMEOUT=1200

# Required if your home directory is on NFS
export NEURON_CC_FLAGS="--temp-dir=/tmp/neuroncc_tmp"
mkdir -p /tmp/neuroncc_tmp

Step 2: Download the model (optional)#

huggingface-cli download \
    Qwen/Qwen3-VL-32B-Instruct \
    --local-dir /path/to/Qwen3-VL-32B-Instruct

Note: This step is optional. You can pass the Hugging Face model ID (e.g. Qwen/Qwen3-VL-32B-Instruct) directly to vllm serve, and the weights will be downloaded automatically on first run.

Step 3: Quantize the text model to MXFP8 (MXFP8 only)#

Skip this step for BF16. Qwen3-VL is not published with MXFP8 weights, so you must quantize the BF16 checkpoint offline. This uses LLM-Compressor and compressed-tensors to apply weight-only, data-free post-training quantization (PTQ) and save a checkpoint in the compressed-tensors format. For background on the quantization workflow, see Quantize using external libraries in the quantization guide.

This is weight-only, data-free PTQ. Run this step on CPU, not on Trainium. Use a machine with enough memory to load the ~64 GB BF16 model, plus disk for both the base and quantized checkpoints. You can quantize once and reuse the checkpoint across Trn3 instances.

Only the text-model linear layers are quantized. The script builds an ignore list that keeps lm_head and every vision-encoder (visual.*) linear layer in full precision — the Neuron MXFP8 path applies to the text transformer only. This text-only scheme mirrors the published Qwen/Qwen3-VL-32B-Instruct-FP8 checkpoint, which likewise quantizes the text model and leaves the vision tower and lm_head in higher precision.

import torch
from compressed_tensors.offload import dispatch_model
from transformers import AutoTokenizer, Qwen3VLForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Path (or Hugging Face ID) of the BF16 base checkpoint.
MODEL_ID = "Qwen/Qwen3-VL-32B-Instruct"
# Where to write the quantized checkpoint.
SAVE_DIR = "/path/to/Qwen3-VL-32B-Instruct-MXFP8-text-only"

# Load the model and tokenizer.
model = Qwen3VLForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Build the ignore list: skip lm_head and every vision-encoder linear layer so
# only the text-model linear layers are quantized to MXFP8.
ignored_list = ["lm_head"]
for name, module in model.named_modules():
    if "visual" in name and isinstance(module, torch.nn.Linear):
        ignored_list.append(name)

print(f"Ignoring {len(ignored_list)} layers from quantization")

# Quantize all remaining Linear layers to MXFP8 via weight-only PTQ.
recipe = QuantizationModifier(targets="Linear", scheme="MXFP8", ignore=ignored_list)
oneshot(model=model, recipe=recipe)

# Confirm the quantized model still generates sane text.
print("========== SAMPLE GENERATION ==============")
dispatch_model(model)
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
print("===========================================")

# Save to disk in compressed-tensors format.
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)
print(f"Saved quantized model to {SAVE_DIR}")

The script prints a short sample generation as a sanity check, then writes the quantized checkpoint and tokenizer to SAVE_DIR. Point the serving commands below at that directory.

Step 4: Run inference#

Run the model through the online serving endpoint or the offline LLM API — choose whichever fits your deployment.

Online serving#

Start a vLLM OpenAI-compatible server.

Note

On a Trn3 instance without EFA (Elastic Fabric Adapter) installed, the server fails to start during EFA-affinity setup — this affects both BF16 and MXFP8. To work around it, prepend NEURON_SKIP_EFA_AFFINITY=1 to the vllm serve command (and set it in the environment before the offline LLM script below).

BF16 (Trn2 or Trn3):

vllm serve /path/to/Qwen3-VL-32B-Instruct \
    --served-model-name Qwen3-VL-32B-Instruct \
    --max-model-len 8192 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --tensor-parallel-size 4 \
    --additional-config '{
        "neuron_config": {
            "quantization": "bf16",
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": true}
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048],
            "vision_attention_block_size": 2048
        }
    }'

MXFP8 (Trn3 only) — point at the quantized checkpoint from Step 3:

vllm serve /path/to/Qwen3-VL-32B-Instruct-MXFP8-text-only \
    --served-model-name Qwen3-VL-32B-Instruct-MXFP8 \
    --max-model-len 8192 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --tensor-parallel-size 4 \
    --block-size 16 \
    --tokenizer-mode slow \
    --no-enable-chunked-prefill \
    --no-enable-prefix-caching \
    --hf-overrides '{"quantization_config": {}}' \
    --additional-config '{
        "neuron_config": {
            "quantization": "mxfp8",
            "modules_to_not_convert": [],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": true}
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048, 4096],
            "vision_attention_block_size": 2048
        }
    }'

The MXFP8-specific settings, compared to the BF16 command:

neuron_config.quantization: "mxfp8" — selects the on-device MXFP8 path. This is a Neuron-specific setting; do not pass vLLM’s --quantization flag.
modules_to_not_convert: [] — an empty list runs the whole text transformer (attention and MLP) in MXFP8. The vision encoder stays BF16 regardless, because its weights were left unquantized in Step 3.
--hf-overrides '{"quantization_config": {}}' — clears the checkpoint’s compressed-tensors quantization_config so the Neuron loader owns the MXFP8 path.
--tokenizer-mode slow, --no-enable-chunked-prefill, --no-enable-prefix-caching, --block-size 16 — pin the deployment to the validated MXFP8 configuration.

Current limitation: at tensor-parallel-size >= 32, change "modules_to_not_convert" to ["mlp"] (keeps the MLP layers in BF16); full MXFP8 ([]) is supported at tensor-parallel-size <= 16.

Once the server is up, send requests using the OpenAI Python SDK (use the --served-model-name you launched with):

import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

# Text-only
response = client.chat.completions.create(
    model="Qwen3-VL-32B-Instruct",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=50,
)
print(response.choices[0].message.content)

# Image + text
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="Qwen3-VL-32B-Instruct",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
        {"type": "text", "text": "Describe this image."},
    ]}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Offline inference#

vLLM-Neuron compiles the model on the first run and caches the artifacts to ~/.cache/vllm/neuron/compile_cache. Subsequent runs skip recompilation and load from cache.

Configuration#

The model has two components, each with its own config object passed via additional_config (see the configuration options reference):

neuron_config: text decoder settings (token/sequence bucket sizes, sampling, quantization).
vision_neuron_config: vision encoder settings (vision token buckets, attention block size, and optional vision TP/DP split).

Bucket sizes control the discrete padded shapes compiled into each NEFF. Each bucket adds compile time; start with one and add more as needed. The two components bucket along different dimensions, so their buckets are configured separately:

num_batched_tokens_buckets (in neuron_config): text-decoder buckets over the number of batched text tokens per forward pass. See compilation options.
num_vision_tokens_buckets (in vision_neuron_config): vision-encoder buckets over the number of vision patches per encoder forward pass (raw T*H*W patches from image_grid_thw, before the 2x2 spatial merge; this is the count select_vision_bucket matches against). The scheduler may batch images from multiple requests into one forward pass, so size the buckets for the total images processed together, not a single request. These scale with image count and resolution:

`num_vision_tokens_buckets`	Approximate capacity
`[2048]`	1–2 images at 448×448 px
`[2048, 8192]`	Up to ~8 images
`[2048, 8192, 20480]`	Up to ~20 images

Run offline inference#

The following script runs text-only, single-image, and multi-image inference. It shows the BF16 LLM(...) construction; for MXFP8, swap in the LLM(...) block that follows (the rest of the script is identical).

import os

os.environ["VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS"] = "1200"
os.environ["NEURON_LIBTORCH_COMPILATION_TIMEOUT"] = "1200"

from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset

MODEL_PATH = "/path/to/Qwen3-VL-32B-Instruct"

# BF16 (Trn2 or Trn3)
llm = LLM(
    model=MODEL_PATH,
    max_model_len=8192,
    max_num_batched_tokens=8192,
    max_num_seqs=4,
    tensor_parallel_size=4,
    additional_config={
        "neuron_config": {
            "quantization": "bf16",
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": True},
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048],
            "vision_attention_block_size": 2048,
        },
    },
)

processor = AutoProcessor.from_pretrained(MODEL_PATH)
sampling_params = SamplingParams(max_tokens=200, temperature=0.0)

# --- Text-only ---
outputs = llm.generate(["What is the capital of France?"], sampling_params)
print(outputs[0].outputs[0].text)

# --- Single image ---
image = ImageAsset("cherry_blossom").pil_image.resize((640, 320))
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate([{"prompt": prompt, "multi_modal_data": {"image": [image]}}], sampling_params)
print(outputs[0].outputs[0].text)

# --- Multi-image ---
images = [
    ImageAsset("stop_sign").pil_image.resize((448, 448)),
    ImageAsset("cherry_blossom").pil_image.resize((448, 448)),
]
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "image"},
    {"type": "text", "text": "Compare these two images."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate([{"prompt": prompt, "multi_modal_data": {"image": images}}], sampling_params)
print(outputs[0].outputs[0].text)

For MXFP8 (Trn3 only), point MODEL_PATH at the quantized checkpoint from Step 3 and build the engine with the MXFP8 config instead — the mirror of the MXFP8 vllm serve command in Step 4:

MODEL_PATH = "/path/to/Qwen3-VL-32B-Instruct-MXFP8-text-only"

llm = LLM(
    model=MODEL_PATH,
    max_model_len=8192,
    max_num_batched_tokens=8192,
    max_num_seqs=4,
    block_size=16,
    tensor_parallel_size=4,
    tokenizer_mode="slow",
    enable_chunked_prefill=False,
    enable_prefix_caching=False,
    hf_overrides={"quantization_config": {}},
    additional_config={
        "neuron_config": {
            "quantization": "mxfp8",
            "modules_to_not_convert": [],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": True},
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048, 4096],
            "vision_attention_block_size": 2048,
        },
    },
)

Vision parallelism (optional)#

By default the vision encoder runs as one DP replica per NeuronCore (TP1), the recommended layout for high-throughput multi-image workloads. To shard the encoder weights across cores for a single large image instead, increase tp_size.

Set tp_size inside vision_neuron_config to change the split. DP is derived automatically as world_size / tp_size, where world_size is your tensor_parallel_size. The commands above use tensor_parallel_size=4, so:

`tp_size`	Vision TP	Vision DP	Best for
`1` (default)	1	4	Multi-image, high throughput
`4`	4	1	Single-image, low latency

The same applies at a larger tensor_parallel_size (e.g. 16), which raises the default vision DP accordingly. Example — TP1, full DP:

"vision_neuron_config": {
    "num_vision_tokens_buckets": [2048],
    "vision_attention_block_size": 2048,
    "tp_size": 1,  # DP = world_size / tp_size
}

Conclusion#

You have deployed Qwen3-VL-32B-Instruct on Trainium — in BF16 on trn2.48xlarge (or Trn3), or in MXFP8 on Trn3 after self-quantizing the text model. The model supports text-only, single-image, multi-image, and video inputs via both the offline LLM API and the OpenAI-compatible online serving endpoint. To validate accuracy after enabling MXFP8, see the accuracy debugging guide. For feature support and accuracy validation results, see the model card.

Next steps#

To tune this deployment for your workload — choosing vision and text sharding with roofline analysis, profiling to find the bottleneck, and enabling the multimodal features and fused kernels — see Optimizing a Vision-Language Model.