This document is relevant for: Trn2, Trn3

Tutorial: Deploy Qwen3-Embedding with vLLM Neuron

This tutorial is a production-ready recipe for deploying Qwen3-Embedding with vLLM Neuron. It covers Qwen3-Embedding-8B in BF16 on Trn2 or Trn3, and presents two deployment paths:

1. Online server: an OpenAI-compatible /v1/embeddings endpoint. The way to serve embeddings to an application.

2. Offline batch: the LLM.embed() Python API in a single process. The higher-throughput path for building an index over a corpus you already have.

For the model’s feature support, accuracy results, and supported checkpoints, see the Qwen3-Embedding model recipe. This tutorial assumes you have already worked through the setup guide and one of the serving quickstarts.

Tested versions

This recipe was validated against the following components. If you are on a newer release, confirm the parameter set still applies before promoting it to production.

Component	Version
Neuron SDK	2.32
vLLM Neuron plugin	Shipped with the Neuron 2.32 release
vLLM (upstream)	The version pinned by the vLLM Neuron plugin for Neuron 2.32
Qwen3-Embedding-8B checkpoint	Qwen/Qwen3-Embedding-8B (Hugging Face)

Choose a deployment

	Online server	Offline batch
Setup	Onevllm serve command	One Python script
Interface	POST /v1/embeddings (OpenAI-compatible)	LLM.embed()
Best for	Applications embedding queries and documents on demand	Bulk index building over a fixed corpus

The two paths are the same model reached through different APIs — the online endpoint is a network wrapper around the offline LLM.embed() call, so both take the same configuration and produce identical vectors. Start with the online server to validate the model and your environment; use the offline path when you have a corpus to embed in one pass.

Prerequisites

• vLLM Neuron environment: A working vLLM Neuron setup on the instance you deploy to. See the setup guide.

• Model access and disk: Access to the Qwen/Qwen3-Embedding-8B checkpoint (~16 GB in BF16). Provide a fast local path via --download-dir or pre-stage the checkpoint.

• Instances: A Trn2 or Trn3 instance with enough NeuronCores for your tensor parallel size — the examples below use --tensor-parallel-size 8, which fits on a trn2.12xlarge. Qwen3-Embedding-8B runs in BF16 on both Trn2 and Trn3.

• Familiarity with vLLM serving: See the online serving quickstart.

What to know before you deploy

Qwen3-Embedding runs as a pooling model, which affects the commands below in a few practical ways:

• Launch with --runner pooling. It is required — the checkpoint declares architectures: ["Qwen3ForCausalLM"], so without it the model loads as a generative model — and it makes the server mount /v1/embeddings.

• Sampling, speculative decoding, and structured-output flags do not apply, since the model emits a vector rather than generated tokens.

• Output vectors are L2-normalized, so on the client a dot product is already the cosine similarity.

For why the pooling path behaves this way (the prefill-only execution model and its consequences), see Pooling models on Trainium.

Deploy an online embedding server

Set environment variables

# Compilation and execution timeouts for Qwen3-Embedding.
export VLLM_NEURON_COMPILATION_TIMEOUT=1200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200

Launch the server

vllm serve Qwen/Qwen3-Embedding-8B \
    --runner pooling \
    --dtype bfloat16 \
    --max-model-len 4096 \
    --tensor-parallel-size 8 \
    --additional-config '{
        "neuron_config": {
            "num_batched_tokens_buckets": [128, 256, 512, 1024, 2048, 4096]
        }
    }'

The server listens on port 8000. On the first launch it compiles the prefill graphs before accepting traffic; subsequent launches reuse cached NEFFs. See the setup guide for cache configuration.

num_batched_tokens_buckets sets the compiled prefill shapes; see bucketing and dynamic shapes for how to size them for your traffic.

For short-sequence workloads with no shared prefix — the common embedding case — add --no-enable-prefix-caching for better performance: disabling it avoids the segmented-prefill overhead. See prefix caching for when it helps.

Validate the online server

# Health check.
curl -i http://localhost:8000/health

# Sample embedding request.
curl http://localhost:8000/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3-Embedding-8B",
        "input": ["What is the capital of France?", "Paris is the capital of France."]
    }'

You should see HTTP/1.1 200 OK on the health check, and an OpenAI-compatible JSON payload with one data[i].embedding array of 4096 floats per input.

Note: For retrieval workloads, prefix each query with a task instruction; embed documents without one. Qwen3-Embedding is trained for the template Instruct: {task description}\nQuery: {query text}, and using it measurably improves retrieval quality over sending the bare query.

Deploy offline batch embedding

For bulk index building, LLM.embed() runs the same model in your own process and avoids per-request HTTP overhead.

import os

os.environ["VLLM_NEURON_COMPILATION_TIMEOUT"] = "1200"
os.environ["VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS"] = "1200"

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen3-Embedding-8B",
    runner="pooling",
    dtype="bfloat16",
    max_model_len=4096,
    tensor_parallel_size=8,
    additional_config={
        "neuron_config": {
            "num_batched_tokens_buckets": [128, 256, 512, 1024, 2048, 4096]
        }
    },
)

docs = [
    "Retrieval-augmented generation grounds answers in retrieved documents.",
    "def add(a, b):\n    return a + b",
    "The quarterly earnings report exceeded analyst expectations by 12%.",
    "Photosynthesis converts sunlight, water, and CO2 into glucose.",
]

for doc, out in zip(docs, llm.embed(docs)):
    vec = out.outputs.embedding
    print(f"dim={len(vec)}  first3={[round(x, 4) for x in vec[:3]]}  | {doc[:40]}")

llm.embed() requires runner="pooling" and raises on a generative runner.

Request a smaller embedding (optional)

Qwen3-Embedding is trained with Matryoshka Representation Learning (MRL), so a truncated prefix of the output vector is still a usable embedding — letting you trade a little quality for a much smaller index. Because the published config.json does not declare that support, opt in at launch:

vllm serve Qwen/Qwen3-Embedding-8B \
    --runner pooling \
    --dtype bfloat16 \
    --max-model-len 4096 \
    --tensor-parallel-size 8 \
    --hf-overrides '{"is_matryoshka": true}'

Then pass dimensions per request. Requests in one batch may ask for different widths:

curl http://localhost:8000/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3-Embedding-8B",
        "input": "What is the capital of France?",
        "dimensions": 256
    }'

The truncated vector is re-normalized, so it remains unit-norm and directly comparable by dot product. For the measured quality-versus-width trade-off, see the model recipe.

Confirm your work

For the online deployment, you are done when the server reports startup complete, /health returns 200, and an embeddings request to port 8000 returns one 4096-float vector per input with L2 norm ≈ 1.0.

For the offline deployment, you are done when llm.embed() returns one embedding per input document, each of the expected dimension.

Either way, sanity-check that the vectors are meaningful, not just well-formed: embed one related pair and one unrelated sentence — for example "What is the capital of France?", "Paris is the capital of France.", and "The mitochondrion is the powerhouse of the cell." — and take the dot product of each pair. The related pair should score visibly higher (typically > 0.6) than the unrelated pair (typically < 0.4). If the two are close, the embeddings are not capturing meaning and something is wrong with the deployment.

Common issues

Compilation hits the timeout

• Possible solution: Raise VLLM_NEURON_COMPILATION_TIMEOUT and VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS before launch. If compilation continues to time out, see the setup guide for cache configuration and shared-cache options.

The server mounts chat endpoints, or /v1/embeddings returns 404

• Possible solution: Add --runner pooling. Qwen3-Embedding-8B declares architectures: ["Qwen3ForCausalLM"], so without that flag the checkpoint is loaded as a generative model.

Startup fails on FP8 KV cache or async scheduling

• Possible solution: An FP8 KV cache is not supported on the embedding path; use the default. Async scheduling is not supported for pooling models; pass --no-async-scheduling.

A dimensions request is rejected

• Possible solution: Launch with --hf-overrides '{"is_matryoshka": true}'. The published checkpoint config does not declare truncation support, so vLLM rejects the parameter until you opt in.

Prefix caching is enabled but throughput does not improve

• Possible solution: The shared prefix is shorter than one whole KV segment, so padding returns the saved tokens to the same compiled shape. Either lengthen the shared prefix or lower the segment size. Note that segmented prefill also pads short requests up to the segment size, so size segments to your traffic rather than to the model’s maximum context.

Latency grows with client concurrency while throughput stays flat

• Possible solution: Expected behavior — prefill processes one request at a time. Raise throughput by running more server replicas behind a load balancer, not by raising concurrency.

Related information

• Qwen3-Embedding model recipe — Feature support, accuracy results, and supported checkpoints.

• Pooling models design — How the pooling execution path is implemented on Neuron.

• Features guide — Feature configuration for Qwen3-Embedding and other models.

• Online serving quickstart — Underlying online serving flow.

• vLLM pooling models — Upstream documentation for --runner pooling.

• vLLM embeddings API — Request and response schema for /v1/embeddings.

This document is relevant for: Trn2, Trn3