This document is relevant for: Trn2, Trn3

Tutorial: Deploy Llama 3.3 70B (FP8) with vLLM Neuron#

This tutorial is a production-ready recipe for deploying Llama 3.3 70B Instruct in static FP8 with vLLM Neuron on a single Trn2 or Trn3 instance using tensor parallelism.

For the model’s feature support, accuracy results, and supported checkpoints, see the Llama 3 model recipe. This tutorial assumes you have already worked through the setup guide and one of the serving quickstarts.

Tested versions#

This recipe was validated against the following components.

Component	Version
Neuron SDK	2.32
vLLM Neuron plugin	Shipped with the Neuron 2.32 release
vLLM (upstream)	The version pinned by the vLLM Neuron plugin for Neuron 2.32
Llama 3.3 70B checkpoint (FP8)	`nvidia/Llama-3.3-70B-Instruct-FP8` (Hugging Face)

Prerequisites#

vLLM Neuron environment: A working vLLM Neuron setup on the instance you deploy to. See the setup guide.
Model access: A Hugging Face account and an access token with read access to the FP8 checkpoint. Fine-grained tokens that lack this permission return HTTP 403 on gated repos.
Disk: The FP8 checkpoint is a large download (~70 GB). Provide a fast local path via --download-dir or pre-stage the checkpoint.
Instance: A trn2.48xlarge or Trn3 instance.
Familiarity with vLLM serving: See the online serving quickstart.

About static FP8#

Llama 3.3 70B runs in static FP8 from a ModelOpt-calibrated checkpoint such as nvidia/Llama-3.3-70B-Instruct-FP8. FP8 roughly halves the weight footprint and speeds up the compute-bound prefill phase, at a small accuracy cost you can quantify from the model recipe.

FP8 is checkpoint-driven: point the server at the FP8 checkpoint and the Neuron backend detects the quantization_config and loads the FP8 weights automatically. There is no FP8-specific launch flag, and no difference between Trn2 and Trn3 — both run the same static FP8 checkpoint. Do not set vLLM’s --quantization flag; the checkpoint’s config selects the path. See FP8 static weight quantization in the features guide for how detection and loading work under the hood.

Deploy on a single instance#

One server serves the whole model with tensor parallelism. Use it for functional validation, development, and serving.

Set environment variables#

# Compilation and execution timeouts for the 70B model.
export VLLM_NEURON_COMPILATION_TIMEOUT=1200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200

# Hugging Face token for the checkpoint.
export HF_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXX

Launch the server#

vllm serve nvidia/Llama-3.3-70B-Instruct-FP8 \
    --tensor-parallel-size 16 \
    --enable-prefix-caching \
    --max-model-len 16384 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 8 \
    --block-size 128 \
    --optimization-level 2 \
    --additional-config '{
        "neuron_config": {
            "kv_segment_size_buckets": [8192],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [8],
            "fp8_packed_kv": true
        }
    }'

The server listens on port 8000. On the first launch it compiles the model graphs before accepting traffic (roughly 15–20 minutes for the 70B model); subsequent launches reuse cached NEFFs. See the setup guide for cache configuration.

Understand the key flags#

--tensor-parallel-size 16 — shards the model across 16 NeuronCores (TP16), the configuration this recipe targets on a single instance.
--max-model-len 16384 — maximum combined prompt + generation length per request.
--max-num-batched-tokens 8192 — the maximum number of tokens processed in a single batched forward pass. Set it explicitly: --enable-prefix-caching turns on chunked prefill, and with chunked prefill enabled vLLM leaves max_num_batched_tokens at its 2048 default rather than raising it to --max-model-len. It must match the last entry of num_batched_tokens_buckets.
num_batched_tokens_buckets: [8192] — the compiled prefill bucket sizes, giving a single, predictable compiled prefill graph. This sizes the prefill graphs; it does not by itself enable segmented prefill or set a segment size (that is kv_segment_size_buckets). The last entry must equal --max-num-batched-tokens.
kv_segment_size_buckets: [8192] — the prefill segment size. Because --max-num-batched-tokens (8192) is below --max-model-len (16384), prefill is segmented: prompts longer than 8192 tokens are processed in 8192-token chunks. Prefix caching also requires segmented prefill, so this entry is what lets --enable-prefix-caching start at all. It must match num_batched_tokens_buckets.
fp8_packed_kv: true — stores the FP8 KV cache in the packed (swizzled) layout. This checkpoint sets kv_cache_quant_algo: FP8, so the KV cache is FP8, and the segmented-prefill attention kernel can only read a packed FP8 K cache. Without this the server fails at startup with NotImplementedError: FP8 segmented prefill requires a packed KV cache.
--max-num-seqs — the maximum number of concurrent sequences (the decode batch size). This is a trade-off between interactivity and throughput: a lower value gives each request more of the instance and lowers per-token latency (better interactivity), while a higher value packs more requests into each decode step for higher aggregate throughput at the cost of higher latency. Start at 8 and tune down for latency-sensitive traffic or up for throughput-oriented batch workloads; keep num_seqs_buckets in the neuron_config in sync with the value you pick.
--block-size 128 — the KV-cache block size in tokens. Neuron defaults block_size to 32; this recipe sets it to 128, the block size validated for the packed FP8 KV path used here (fp8_packed_kv: true with segmented prefill). kv_segment_size_buckets must stay divisible by block_size, so 128 divides the 8192-token segment bucket cleanly.
--optimization-level 2 — raises the compiler optimization level from Neuron’s O1 default to O2 for better serving performance on the 70B graphs. Neuron defaults to O1 (vLLM’s own default is O2); passing this flag opts back into the higher optimization level. Expect a somewhat longer first compile in exchange.

Validate the server#

# Health check.
curl -i http://localhost:8000/health

# Sample completion.
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "nvidia/Llama-3.3-70B-Instruct-FP8",
        "prompt": "The capital of France is ",
        "max_tokens": 16
    }'

You should see HTTP/1.1 200 OK on the health check, and an OpenAI-compatible JSON payload with a coherent choices[0].text on the completion.

Add EAGLE3 speculative decoding (optional)#

Llama 3 supports EAGLE3 speculative decoding to raise decode throughput. Pair the target with a matching EAGLE3 draft — for example RedHatAI/Llama-3.3-70B-Instruct-speculator.eagle3 — via --speculative-config:

vllm serve nvidia/Llama-3.3-70B-Instruct-FP8 \
    --tensor-parallel-size 16 \
    --max-model-len 16384 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 8 \
    --block-size 128 \
    --optimization-level 2 \
    --additional-config '{
        "neuron_config": {
            "kv_segment_size_buckets": [8192],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [8],
            "fp8_packed_kv": true
        }
    }' \
    --speculative-config '{
        "method": "eagle3",
        "model": "RedHatAI/Llama-3.3-70B-Instruct-speculator.eagle3",
        "num_speculative_tokens": 3
      }'

The neuron_config block carries over unchanged from the launch command above — the FP8 KV cache requirements apply here too. Expect a longer first compile, since both the target and the draft model are compiled.

Under greedy sampling EAGLE3 is lossless — the target output is unchanged. For the full walkthrough, including how to read the acceptance rate and tune num_speculative_tokens, see the EAGLE3 speculative decoding tutorial.

Confirm your work#

You are done when the server reports startup complete, /health returns 200, and a completion request to port 8000 returns a coherent OpenAI-compatible response.

Common issues#

Compilation hits the timeout#

Possible solution: Llama 3.3 70B is large; cold compilation can exceed the default 600-second budget (typically 15–20 minutes). Raise VLLM_NEURON_COMPILATION_TIMEOUT before launch. If compilation continues to time out, see the setup guide for cache configuration and shared-cache options.

Out of memory during compilation or serving#

Possible solution: Reduce --max-model-len or --max-num-seqs (keeping num_seqs_buckets in sync). If you lower --max-num-batched-tokens, lower the last entry of num_batched_tokens_buckets to match, or startup validation fails.

HTTP 403 when the server pulls the model#

Possible solution: Your Hugging Face token lacks read access to the checkpoint repository. Update the token at https://huggingface.co/settings/tokens and restart the server.

Tutorial: Deploy Llama 3.3 70B (FP8) with vLLM Neuron

Contents