This document is relevant for: Trn2, Trn3

Qwen3-32B Model Recipe#

Introduction#

Qwen3-32B is a dense decoder-only language model from the Qwen family. In vLLM Neuron, the dense 32B path supports the published BF16 checkpoint and a native MXFP8 path for compressed-tensors checkpoints on Trn3.

Qwen3-32B is supported for inference serving with vLLM Neuron Plugin using the Neuron SDK on AWS Trainium2 (trn2) and Trainium3 (trn3) hardware.

Compatible model checkpoints:

Model	Checkpoint	Hardware	Quantization
Qwen3-32B	Qwen/Qwen3-32B	Trn2, Trn3	BF16
Qwen3-32B (native MXFP8)	Built locally from `Qwen/Qwen3-32B` with `examples/vllm_neuron/models/qwen3/build_mxfp8_checkpoint.py`	Trn3	MXFP8

The repository includes run.py for the BF16 path and build_mxfp8_checkpoint.py plus run_mxfp8.py for the native MXFP8 path. MXFP8 is Trn3-only.

Features#

Per-model feature availability for Qwen3-32B dense. See the features guide for configuration details.

Category	Feature	Status
Inputs	Text	✅
Quantization	BF16	✅
	MXFP8 weights (Trn3)	✅
	KV cache FP8 — packed swizzled layout (Trn3, MXFP8)	✅ (V1)
Parallelism	Tensor parallelism (TP)	✅
	Replica data parallelism (DP)	✅
	Pipeline parallelism (PP)	❌
	Sequence parallelism (SP) — prefill	✅
Performance	Continuous batching	✅
	Segmented prefill	✅
	Prefix caching (APC)	✅
	On-device sampling (greedy, top-k, top-p)	✅
	Async scheduling	✅
Serving	Structured outputs / tool-calling enforcement	✅ (V1)
	Disaggregated inference (DI), MXFP8 packed FP8 KV	✅ (V1)
	EAGLE3 speculative decoding, single server	✅ (V1)
Compilation	torch.compile (native `neuron_native` and XLA `vllm_neuron` backends)	✅
	CPU mode (testing)	✅

Status legend:

✅ Supported: integrated and tested for Qwen3-32B dense
✅ (V1) Supported: requires Model Runner V1 (MRv1) in this release
❌ Not supported: may be considered for future releases

The plugin selects one model runner per server from the complete configuration. By default, it automatically selects MRv1 for the rows marked (V1). Common features, such as TP, continuous batching, and async scheduling, also work on MRv1.

The DI configuration in this recipe uses MRv1 on both the prefill and decode servers because it uses packed FP8 KV. CPU tests with enforce_eager also select MRv1. Feature support does not guarantee support for every combination. See the runner selection guide and the validated deployment configurations.

Deployment configurations#

Use the deployment tutorial to prepare a checkpoint and launch a TP8 single server. The tutorial also describes a packed-FP8-KV 1P1D deployment with TP4/LNC2 prefill and TP8/LNC1 decode at bs64.

Weight format	Hardware	Checkpoint
BF16	Trn2, Trn3	`Qwen/Qwen3-32B`
Native MXFP8	Trn3	Convert the BF16 source with `build_mxfp8_checkpoint.py`

Checkpoint provenance#

Pin the BF16 checkpoint to an immutable Hugging Face revision. For MXFP8, record the source revision, converter commit, conversion options, dependency versions, and checksums. Keep lm_head in BF16. The tutorial shows the download, conversion, and checksum commands.

Tensor parallelism and KV cache#

BF16 weights use BF16 KV. The BF16 baseline uses TP8, and BF16 examples and accuracy tests cover TP8, TP16, and TP32. MXFP8 weights are validated with both BF16 KV and packed FP8 KV. MXFP8 tests cover TP8 single-server inference and TP4/LNC2 prefill → TP8/LNC1 decode through real NIXL transfer. The native MXFP8 example (run_mxfp8.py) uses tensor_parallel_size=8, max_model_len=16384, and num_batched_tokens_buckets=[8192].

The MXFP8 performance grid includes TP4/DP16, TP8/DP8, TP16/DP4, and TP32/DP2. The DI target uses TP4 for prefill and TP8 for decode, with packed FP8 KV on both roles. Prefill uses O2; decode uses O3, block size 128, and a pinned 5232-block cache.

Use the same backend, TP/DP, LNC, optimization level, context buckets, and KV dtype when comparing accuracy or performance. Keep per-replica batch and node concurrency distinct when sizing a DP deployment.

Component data parallelism also selects MRv1. See the runner selection guide for details.

Accuracy Evaluation#

Logit checks compare greedy device output with a Hugging Face reference. GSM8K and LongBench-v2 provide task-level checks. The table lists the existing CI coverage and acceptance floors. These floors are test criteria, not measured accuracy scores.

Evaluation	Existing coverage	CI floor
BF16 GSM8K	TP8/16/32, bs8, 50 examples, max length 10,240	0.85
MXFP8 GSM8K	TP8/DP1, 200 examples. Packed FP8 KV: bs8 at 32k, bs64/128 at 10,240. BF16 KV: bs8 at 32k	0.94
BF16 LongBench-v2	TP8, bs8, 32k	0.378
MXFP8 LongBench-v2	TP8, bs8, 32k, BF16 KV	0.378
Greedy logits	BF16 TP8/16/32; MXFP8 TP8 with BF16 and packed FP8 KV; MXFP8 DI	Per-target numerical tolerances

Mixed-LNC DI accuracy measurements#

These measurements use MXFP8, TP4/LNC2 prefill, TP8/LNC1 decode, packed FP8 KV, and real NIXL transfer. GSM8K evaluates 200 examples with flexible-extract scoring.

Evaluation	Native `neuron_native`	XLA `vllm_neuron`	Method
Greedy logits	Pass	Pass	Device logits compared with the Hugging Face reference
GSM8K	98.0%	97.0%	200 examples; 94% acceptance floor

The logit and GSM8K measurements use pre-release builds of the Neuron 2.33 vLLM Neuron plugin. These are measured accuracy snapshots. Use the same checkpoint and serving settings to reproduce these measurements.

Both evaluations use a maximum sequence count of 64, the 5232-block cache allocation, and a maximum model length of 10,240.

When reporting a measured score, include the dataset revision, sample count, scoring method, checkpoint identity, and software versions. Record the backend, TP/DP, LNC, batch, context length, and KV dtype with the result.

Common Issues#

Issue	Cause	Resolution
Native MXFP8 TP64 is not supported	The native MXFP8 MLP scale layout requires `intermediate_size / tensor_parallel_size` to be a multiple of `32`. For Qwen3-32B, `25600 / 64 = 400`, so the per-rank MLP intermediate does not satisfy the MX scale-packing contract.	Use TP8, TP16, or TP32 for native MXFP8. The BF16 path does not have this specific restriction.
No published Qwen3-32B MXFP8 checkpoint	The dense MXFP8 path serves a self-quantized compressed-tensors checkpoint rather than a public upstream MXFP8 release.	Generate the checkpoint locally with `examples/vllm_neuron/models/qwen3/build_mxfp8_checkpoint.py`, then serve it with `run_mxfp8.py` or `vllm serve ... --additional-config '{"neuron_config":{"quantization":"mxfp8"}}'`.
Native MXFP8 on Trn2 is not supported	The native MXFP8 path depends on Trn3 support.	Use BF16 on Trn2, or move the MXFP8 deployment to Trn3.

Qwen3-32B Model Recipe

Contents