This document is relevant for: Trn2, Trn3

Tutorial: Deploy gpt-oss with vLLM Neuron

This tutorial is a production-ready recipe for deploying gpt-oss with vLLM Neuron. It covers gpt-oss 20B and gpt-oss 120B, MXFP4-quantized on Trn3 (both models also run on Trn2 in BF16), and presents two deployment paths:

1. Single instance (non-DI): one server with tensor parallelism. The easiest way to stand gpt-oss up.

2. Disaggregated inference (DI): prefill and decode served on separate instances. More setup, but yields the best performance under load.

For the model’s feature support, accuracy results, and supported checkpoints, see the gpt-oss model recipe. This tutorial assumes you have already worked through the setup guide and one of the serving quickstarts.

Tested versions

This recipe was validated against the following components. If you are on a newer release, confirm the parameter set still applies before promoting it to production.

Component	Version
Neuron SDK	2.31
vLLM Neuron plugin	Shipped with the Neuron 2.31 release
vLLM (upstream)	The version pinned by the vLLM Neuron plugin for Neuron 2.31
gpt-oss 20B checkpoint	openai/gpt-oss-20b (Hugging Face)
gpt-oss 120B checkpoint	openai/gpt-oss-120b (Hugging Face)

Choose a deployment

	Single instance (non-DI)	Disaggregated inference (DI)
Setup	One server, one command	Separate prefill + decode instances, a proxy, and EFA networking
Best for	Functional validation, development, light or non-latency-critical traffic	Production and stress/benchmark workloads
Performance	Adequate for simple tasks	Best throughput and latency under load

Start with the single-instance deployment to validate the model and your environment. Move to disaggregated inference when you need the best performance — only the DI setup delivers the full throughput and latency of gpt-oss on Neuron.

Prerequisites

• vLLM Neuron environment: A working vLLM Neuron setup on each instance you deploy to. See the setup guide.

• Model access and disk: Access to the gpt-oss checkpoint you intend to serve. gpt-oss 120B is a large download; provide a fast local path via --download-dir or pre-stage the checkpoint.

• Instances: On Trn3, both gpt-oss 20B and 120B use MXFP4 weights (auto-selected) — the path this recipe targets. Both models also run on Trn2 in BF16. The disaggregated deployment additionally requires two instances with EFA connectivity between them (see that section).

• Familiarity with vLLM serving: See the online serving quickstart. For the disaggregated path, also work through the disaggregated inference tutorial.

Deploy on a single instance (non-DI)

This is the simplest path: one server serving the whole model with tensor parallelism. Use it for functional validation, development, and light workloads.

Set environment variables

# Compilation and execution timeouts for gpt-oss.
export VLLM_NEURON_COMPILATION_TIMEOUT=1200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200

Launch the server

vllm serve openai/gpt-oss-120b \
    --tensor-parallel-size 8 \
    --enable-prefix-caching \
    --no-disable-hybrid-kv-cache-manager \
    --max-model-len 16384 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --hf-overrides '{"quantization_config": {}}' \
    --additional-config '{
        "neuron_config": {
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4]
        }
    }'

The Neuron backend auto-selects the MXFP4 weights on Trn3. To serve gpt-oss 20B instead, swap the model identifier to openai/gpt-oss-20b; the same command works on Trn3 (MXFP4) or Trn2 (BF16).

The server listens on port 8000. On the first launch it compiles the model graphs before accepting traffic; subsequent launches reuse cached NEFFs. See the setup guide for cache configuration.

Validate the single-instance server

# Health check.
curl -i http://localhost:8000/health

# Sample completion (swap in openai/gpt-oss-20b if you launched 20B).
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "openai/gpt-oss-120b",
        "prompt": "The capital of France is ",
        "max_tokens": 16
    }'

You should see HTTP/1.1 200 OK on the health check, and an OpenAI-compatible JSON payload with a coherent choices[0].text on the completion.

Deploy with disaggregated inference (best performance)

Disaggregated inference separates prompt processing (prefill) from token generation (decode) so you can size and scale each phase independently, which is what delivers the best throughput and latency for gpt-oss. This section pins the tested topology and parameter set; for the mechanics of DI on Neuron — the NIXL KV connector, the proxy router, and how to scale from 1P1D to xPyD — see the disaggregated inference tutorial and the disaggregated inference design.

Target configuration

This deploys a 1P1D topology: one prefill instance (kv_producer) and one decode instance (kv_consumer), fronted by a proxy router. KV cache is transferred from prefill to decode over NIXL using the LIBFABRIC backend (EFA on AWS). Both instances serve MXFP4 weights.

The prefill configuration is identical for both models. The decode configuration differs only in its data-parallel degree, which sets the expert-parallel width and the global decode batch size.

Role	Parallelism	max_num_seqs	Global decode batch
Prefill (20B and 120B)	attention TP4 (DP1), MoE EP2	1	—
Decode, gpt-oss 20B	TP8 DP4 EP32 (1 expert/rank)	4	16 (4 × DP4)
Decode, gpt-oss 120B	TP8 DP8 EP64 (2 experts/rank)	4	32 (4 × DP8)

max_num_seqs is the batch bucket per data-parallel replica, so the global decode batch is max_num_seqs × data-parallel degree — for the 120B TP8 DP8 EP64 setup above, 4 × 8 = 32. The max_num_seqs 4 in this recipe is a conservative starting point. For a throughput-optimized deployment, raise max_num_seqs so the global decode batch reaches 128 or more (for 120B at DP8, that is max_num_seqs 16), then confirm the larger batch fits in HBM at your max_model_len.

Parameters shared by all roles:

Setting	Value	Notes
Quantization	MXFP4	Pass --dtype bfloat16 and neuron_config.quantization: "mxfp4"; the Neuron backend loads the published MXFP4 weights.
max_model_len	16384	Set explicitly on every role.
max_num_batched_tokens	8192	Prefill segment size; matches the single kv-segment and batched-tokens bucket.
Bucketing	kv_segment_size_buckets: [8192], num_batched_tokens_buckets: [8192]	One bucket per role for a single, predictable compiled graph.
Prefix caching	On (default)	See prefix caching benchmark tutorial for tuning.
On-device sampling	top-k (default)	Set all_greedy: true in on_device_sampling_config for deterministic outputs.

DI prerequisites

• Two instances: One for prefill and one for decode, each with all 64 NeuronCores available.

• EFA connectivity between the instances: DI transfers KV cache over NIXL on the LIBFABRIC backend. The two instances must reach each other over EFA. See the disaggregated inference tutorial for network setup.

• The checkpoint staged on both instances.

1. Set environment variables on both instances

gpt-oss compilation and multi-node startup run longer than the vLLM defaults. Set these on both the prefill and decode instances before launching.

# Compilation and startup timeouts for gpt-oss.
export VLLM_NEURON_COMPILATION_TIMEOUT=2400
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200
export VLLM_ENGINE_READY_TIMEOUT_S=5400

# NIXL side channel: listen on all interfaces so the peer can reach it over EFA.
export VLLM_NIXL_SIDE_CHANNEL_HOST=0.0.0.0

2. Launch the decode server

On the decode instance, bind all 64 cores and set the decode-side NIXL side channel port. This is the kv_consumer.

export NEURON_VISIBLE_DEVICES="0-63"
export VLLM_NIXL_SIDE_CHANNEL_PORT=5659

vllm serve openai/gpt-oss-120b \
    --tensor-parallel-size 8 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --optimization-level 2 \
    --dtype bfloat16 \
    --max-model-len 16384 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --max-logprobs 0 \
    --no-disable-hybrid-kv-cache-manager \
    --port 8200 \
    --hf-overrides '{"quantization_config": {}}' \
    --kv-transfer-config '{"kv_connector": "NixlConnector", "kv_role": "kv_consumer", "kv_buffer_device": "cuda", "kv_connector_extra_config": {"backends": ["LIBFABRIC"]}}' \
    --additional-config '{
        "neuron_config": {
            "quantization": "mxfp4",
            "embedding_dp_size": 8,
            "lm_head_dp_size": 8,
            "kv_segment_size_buckets": [8192],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4]
        }
    }'

For gpt-oss 20B, serve openai/gpt-oss-20b and drop the decode-side data-parallel degree to 4 (--data-parallel-size 4, embedding_dp_size: 4, lm_head_dp_size: 4, and omit --optimization-level 2). This yields TP8 DP4 EP32 and a global decode batch of 16.

The decode server reports Application startup complete on port 8200 once it is ready. It has no dependency on the prefill server at startup, so you can launch the prefill server (next step) at the same time rather than waiting for decode to finish; the proxy in step 4 is the only component that requires both to be up.

3. Launch the prefill server

On the prefill instance, bind four cores and set the prefill-side NIXL side channel port. This is the kv_producer. The prefill configuration is the same for both models — only the model identifier changes.

export NEURON_VISIBLE_DEVICES="0-3"
export VLLM_NIXL_SIDE_CHANNEL_PORT=5559

vllm serve openai/gpt-oss-120b \
    --tensor-parallel-size 4 \
    --data-parallel-size 1 \
    --enable-expert-parallel \
    --dtype bfloat16 \
    --max-model-len 16384 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 1 \
    --no-disable-hybrid-kv-cache-manager \
    --port 8100 \
    --hf-overrides '{"quantization_config": {}}' \
    --kv-transfer-config '{"kv_connector": "NixlConnector", "kv_role": "kv_producer", "kv_buffer_device": "cuda", "kv_connector_extra_config": {"backends": ["LIBFABRIC"]}}' \
    --additional-config '{
        "neuron_config": {
            "quantization": "mxfp4",
            "ep_degree": 2,
            "kv_segment_size_buckets": [8192],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [1]
        }
    }'

The prefill server reports Application startup complete on port 8100 once it is ready.

Note

On the first launch each server compiles the model graphs before it accepts traffic; this is why the timeouts in step 1 are raised. Subsequent launches reuse cached NEFFs unless you invalidate the cache. See the setup guide for cache configuration.

Scale prefill to fully utilize the instance

A single TP4 prefill server uses only 4 of the instance’s 64 cores, leaving the other 60 idle and capping end-to-end throughput. For best performance, tile up to 16 TP4 prefill servers across those 64 cores to match one TP8 DP8 EP64 decode server — a 16P1D topology that leaves no idle cores. Because prefill and decode scale independently under DI, adding prefill servers does not change the decode topology.

Launch each additional prefill server with the same command above, but give each one a distinct 4-core slice via NEURON_VISIBLE_DEVICES so the servers do not contend for the same cores, along with a unique HTTP port and a unique VLLM_NIXL_SIDE_CHANNEL_PORT:

# Prefill server 0
export NEURON_VISIBLE_DEVICES="0-3"
export VLLM_NIXL_SIDE_CHANNEL_PORT=5559
# vllm serve ... --port 8100

# Prefill server 1
export NEURON_VISIBLE_DEVICES="4-7"
export VLLM_NIXL_SIDE_CHANNEL_PORT=5560
# vllm serve ... --port 8101

# ... and so on through server 15 (NEURON_VISIBLE_DEVICES="60-63", port 8115)

Then list every prefill port on the proxy in the next step, and it round-robins requests across them:

    --prefiller-host PREFILL_HOST --prefiller-port 8100 8101 8102 ...

Run these as 16 separate prefill servers, not one server with a data-parallel degree of 16. With expert parallelism enabled, raising the data-parallel degree turns the MoE layers into a cross-DP expert-parallel sharding — a different (and here undesirable) expert layout. Keeping each server at TP4 DP1 preserves the tested prefill sharding and simply replicates it.

For the general xPyD scaling procedure, see the disaggregated inference tutorial.

4. Launch the proxy router

Launch the proxy only after both the decode and prefill servers have reported Application startup complete. The proxy accepts client requests and coordinates the prefill → decode handoff. Run it on the prefill instance (or any instance that can reach both). Replace PREFILL_HOST and DECODE_HOST with the instances’ addresses.

python3 examples/vllm_neuron/vllm/disaggregated_inference/toy_proxy_server.py \
    --port 8000 \
    --host 0.0.0.0 \
    --prefiller-host PREFILL_HOST --prefiller-port 8100 \
    --decoder-host DECODE_HOST --decoder-port 8200

The proxy listens on port 8000 and routes each request to the prefill server for prompt processing, then to the decode server, which pulls the KV cache from prefill over NIXL and streams tokens back. toy_proxy_server.py ships in the vLLM Neuron repository. See the disaggregated inference tutorial for details on the proxy and the request lifecycle.

If you launched multiple prefill servers (see Scale prefill to fully utilize the instance), pass all their ports to --prefiller-port.

5. Understand the gpt-oss-specific flags

A few flags warrant explicit explanation because they are gpt-oss specific, DI specific, or because the defaults are not what you want for this recipe.

• --enable-expert-parallel with ep_degree / data parallelism — gpt-oss is a Mixture-of-Experts model. On decode, expert parallelism is derived from the tensor- and data-parallel degrees (TP8 × DP4 = EP32 for 20B; TP8 × DP8 = EP64 for 120B). On prefill, ep_degree: 2 sets the MoE expert-parallel width.

• embedding_dp_size / lm_head_dp_size — set to the decode data-parallel degree so the embedding and LM-head layers are replicated across the DP replicas.

• --max-logprobs 0 (decode) — disables logprob support so the decode graph skips gathering the full logits tensor before sampling, which saves decode time and memory. Set it only on the decode server; leave it off if your workload requests logprobs.

• --optimization-level 2 (120B decode) — raises the compiler optimization level for the larger model’s decode graph.

6. Validate the deployment

Send requests to the proxy (port 8000), not to the prefill or decode servers directly. With all three processes running, confirm the response shape.

# Health check against the proxy.
curl -i http://PREFILL_HOST:8000/health

You should see HTTP/1.1 200 OK.

Next, send a sample completion request. Replace openai/gpt-oss-120b with openai/gpt-oss-20b if you launched the 20B variant.

curl http://PREFILL_HOST:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "openai/gpt-oss-120b",
        "prompt": "The capital of France is ",
        "max_tokens": 16
    }'

The response is an OpenAI-compatible JSON payload. Confirm the following:

• choices[0].text contains a coherent continuation of the prompt.

• choices[0].finish_reason is length (because the request hit max_tokens) or stop.

• usage.prompt_tokens and usage.completion_tokens are populated.

Note

With default top-k sampling, completion content varies across runs. For deterministic outputs, add "on_device_sampling_config": {"all_greedy": true} to the neuron_config on both roles. For substantive accuracy validation, see the accuracy debugging guide.

7. Benchmark and tune prefix caching

Before promoting this configuration to production, run a workload-shaped benchmark. The companion tutorial uses gpt-oss as its subject, so its setup, datasets, and tuning guidance apply directly to this recipe.

• Prefix caching benchmark tutorial — Quantify the throughput and time-to-first-token impact of prefix caching for your prompt distribution and tune accordingly.

Confirm your work

For the single-instance deployment, you are done when the server reports startup complete, /health returns 200, and a completion request to port 8000 returns a coherent OpenAI-compatible response.

For the disaggregated deployment, you have a successful deployment when:

• The decode and prefill servers each report Application startup complete, and the proxy’s /health endpoint returns 200.

• A completion request sent through the proxy returns an OpenAI-compatible response on the validation prompt with the expected shape.

• Server logs show the decode server pulling KV cache from prefill over NIXL / LIBFABRIC (no NIXL handshake errors).

• A short representative-traffic run produces stable, on-spec completions, and throughput and latency on your benchmark traffic meet the targets you set.

Common issues

Compilation hits the timeout

• Possible solution: gpt-oss 120B is large; cold compilation can exceed the default 600-second budget. Raise VLLM_NEURON_COMPILATION_TIMEOUT (and, for the disaggregated path, VLLM_ENGINE_READY_TIMEOUT_S) before launch. If compilation continues to time out, see the setup guide for cache configuration and shared-cache options.

NIXL handshake fails or the decode server cannot reach prefill (DI)

• Possible solution: Confirm VLLM_NIXL_SIDE_CHANNEL_HOST=0.0.0.0 on both roles, that the prefill and decode side-channel ports differ (5559 and 5659 in this recipe), and that the two instances can reach each other over EFA.

Degenerate or garbled decode output (DI)

• Possible solution: Keep the decode topology at the tested expert-parallel degree for the model (EP32 for 20B, EP64 for 120B). Changing the data-parallel degree changes the expert-parallel width, and untested EP degrees have produced degenerate output. Match embedding_dp_size and lm_head_dp_size to --data-parallel-size.

Related information

• gpt-oss model recipe — Feature support, accuracy results, and supported checkpoints for gpt-oss.

• Disaggregated inference tutorial — DI mechanics: NIXL KV connector, proxy router, and 1P1D → xPyD scaling.

• Disaggregated inference design — How DI control and data flow are implemented on Neuron.

• Prefix caching benchmark tutorial — Benchmark prefix caching with gpt-oss across open-source datasets.

• Features guide — Feature configuration for gpt-oss and other models.

• Online serving quickstart — Underlying online serving flow.

• Accuracy debugging guide — Investigate accuracy issues if validation completions look wrong.

This document is relevant for: Trn2, Trn3