This document is relevant for: Trn2, Trn3

Qwen3-Embedding Model Recipe#

Introduction#

Qwen3-Embedding is an open-weight text-embedding model family released by the Qwen team, built on the Qwen3 backbone and designed for semantic search, retrieval, RAG, clustering, and deduplication. Given a text input it returns a single fixed-size vector instead of generated text, so each request runs one forward pass over the prompt.

Qwen3-Embedding is supported for inference serving with vLLM using the Neuron SDK on AWS Trainium2 (trn2) and Trainium3 (trn3) hardware.

Compatible model checkpoints:

Model	HuggingFace	Hardware	Embedding dim	Quantization	Validated
Qwen3-Embedding-8B	Qwen/Qwen3-Embedding-8B	Trn2, Trn3	4096	BF16	✅
Qwen3-Embedding-4B	Qwen/Qwen3-Embedding-4B	Trn2, Trn3	2560	BF16	❌
Qwen3-Embedding-0.6B	Qwen/Qwen3-Embedding-0.6B	Trn2, Trn3	1024	BF16	❌

The 4B and 0.6B checkpoints share the same Qwen3 architecture as the 8B and are expected to work through the same pooling path, but have not yet been validated on Neuron; the accuracy and performance results below are for the 8B.

Embedding serving requires --runner pooling. All three checkpoints declare architectures: ["Qwen3ForCausalLM"] in their config.json — the same value as the generative Qwen3 models — so without that flag the checkpoint is loaded as a generative model.

Features#

Per-model feature availability for Qwen3-Embedding. See the features guide for configuration details and the cross-model feature compatibility matrix.

Category	Feature	Status
Inputs	Text	✅
Tasks	Embeddings (`/v1/embeddings`, `LLM.embed()`)	✅
	Classification / scoring / reranking	❌
Quantization	BF16 weights	✅
	FP8 / MXFP4 weights	❌
	KV cache FP8	❌
Parallelism	Tensor parallelism (TP)	✅
	Data parallelism (DP)	✅
	Pipeline parallelism (PP)	❌
	Context parallelism (CP)	❌
Performance	Segmented prefill	✅
	Prefix caching (APC)	✅
	Matryoshka (variable output dimensions)	✅
	Async scheduling	❌
	Disaggregated inference (1P1D / xPyD)	❌
Compilation	torch.compile (XLA backend)	✅
	CPU mode (testing)	✅

Status legend:

✅ Supported: integrated and tested for Qwen3-Embedding
❌ Not supported: may be considered for future releases

The deployment tutorial walks through deploying Qwen3-Embedding-8B via two paths: an online OpenAI-compatible /v1/embeddings server and offline batch embedding with LLM.embed().

Because an embedding model emits a vector rather than generating tokens, there is no decode phase. Features that act on generated tokens — sampling, speculative decoding, structured outputs — do not apply, and the decode graph is neither compiled nor warmed up, which shortens startup.

Tensor parallelism is validated at tensor_parallel_size ∈ {2, 4, 8, 16, 32} on both Trn2 and Trn3, and in combination with data parallelism.

Prefix caching requires segmented prefill: set --max-num-batched-tokens to one of {512, 1024, 2048, 4096, 8192} and below --max-model-len. Enabling it with single-shot prefill fails at startup with an explanatory error. It benefits workloads whose requests share a long leading prefix, such as a fixed task instruction, a RAG template head, or a constant document preamble. See prefix caching for configuration details.

Matryoshka support lets a client request a shorter, re-normalized vector by passing dimensions per request. Because Qwen3-Embedding-8B’s published config.json does not declare Matryoshka support, opt in at launch with --hf-overrides '{"is_matryoshka": true}'.

Accuracy Evaluation#

Accuracy measured on real hardware with BF16 weights (Trn2), using the MTEB harness with the official per-task Qwen3 query instruction.

Benchmark	Neuron (BF16)	Published
STS12 (Spearman)	0.8639	0.8614
NFCorpus (NDCG@10)	0.4143	0.4145
SciFact (NDCG@10)	0.7839	0.7846

The Published column is the Qwen3-Embedding-8B score from the MTEB results repository ; our BF16-on-Neuron scores match it to within 0.001.

Requesting a truncated embedding trades a small amount of quality for a much smaller vector. Measured on SciFact (NDCG@10), relative to the full 4096 dimensions:

Output dimensions	32	64	128	256	2048	4096
NDCG@10	0.542	0.669	0.736	0.752	0.783	0.784
% of full width	69%	85%	94%	96%	99.9%	100%

Tutorials#

Tutorial: Deploy Qwen3-Embedding-8B with vLLM Neuron — End-to-end online (/v1/embeddings) and offline (LLM.embed()) embedding deployment recipe, including variable output dimensions and prefix caching.