This document is relevant for: Trn2, Trn3

Llama 3 Model Recipe#

Introduction#

Llama 3 is a family of open-weight, decoder-only text language models from Meta, instruction-tuned for chat and general task-following. This recipe covers three sizes across the Llama 3.1 / 3.2 / 3.3 releases — a 1B, an 8B, and a 70B — which share the same architecture and serving path and differ only in scale and the parallelism degree used to serve them.

Llama 3 is supported for inference serving with vLLM using the Neuron SDK on AWS Trainium2 (trn2) and Trainium3 (trn3) hardware.

Compatible model checkpoints:

Model	HuggingFace	Hardware	Quantization
Llama 3.2 1B Instruct	meta-llama/Llama-3.2-1B-Instruct	Trn2, Trn3	BF16, FP8 (static)
Llama 3.1 8B Instruct	meta-llama/Llama-3.1-8B-Instruct	Trn2, Trn3	BF16, FP8 (static)
Llama 3.3 70B Instruct	meta-llama/Llama-3.3-70B-Instruct	Trn2, Trn3	BF16, FP8 (static)

The three sizes are byte-identical in serving path; only the tensor-parallel degree changes. All three run in BF16 out of the box, and all three accept a static FP8 checkpoint (see below). There is no quantization difference between Trn2 and Trn3 — both run the same static FP8 checkpoints.

Quantization#

Llama 3 supports tensor-wide (per-tensor) static FP8 checkpoints in the NVIDIA ModelOpt format, for example nvidia/Llama-3.3-70B-Instruct-FP8. The checkpoint carries per-tensor static weight, activation, and KV-cache scales in its quantization_config (quant_method: "modelopt", quant_algo: "FP8"). The Neuron backend detects the format and loads the FP8 weights automatically — you do not set vLLM’s --quantization flag. See the FP8 static weight quantization section of the features guide for details.

Both Trn2 and Trn3 run the same static FP8 checkpoints. Internally, Trn3 routes FP8 matmuls through STATIC_MX kernels for a prefill speedup, but the checkpoint and the way you launch the server are identical on both platforms.

Speculative decoding#

Llama 3 supports EAGLE3 speculative decoding. Pair a target checkpoint with a matching EAGLE3 draft, for example RedHatAI/Llama-3.3-70B-Instruct-speculator.eagle3 as the draft for meta-llama/Llama-3.3-70B-Instruct. Under greedy sampling EAGLE3 is a lossless acceleration — the target model’s output is unchanged. See the EAGLE3 speculative decoding tutorial for an end-to-end walkthrough and acceptance-rate tuning.

Features#

Per-model feature availability for Llama 3. See the features guide for configuration details.

Category	Feature	Status	Runner
Inputs	Text	✅	MRv2
Quantization	BF16 weights	✅	MRv2
	FP8 static (per-tensor, ModelOpt)	✅	MRv2
	KV cache FP8	✅	MRv2
Parallelism	Tensor parallelism (TP)	✅	MRv2
	Data parallelism (DP)	✅ (see Known issues)	MRv2
	Expert parallelism (EP)	N/A	n/a
	Pipeline parallelism (PP)	❌	n/a
Performance	Continuous batching	✅	MRv2
	Segmented prefill	✅	MRv2
	Prefix caching (APC)	✅	MRv2
	Speculative decoding (EAGLE3)	✅	MRv1
	Speculative decoding (DFlash)	✅	MRv1
	Disaggregated inference (1P1D / xPyD)	✅	MRv2
	On-device sampling (greedy, top-k, top-p)	✅	MRv2
	Async scheduling	✅	MRv2
Serving	Structured outputs / tool calling	✅	MRv1
Compilation	torch.compile (XLA backend)	✅	MRv2
	CPU mode (testing)	✅	MRv2 (MRv1 with `enforce_eager`)

Status legend:

✅ Supported: integrated and tested for Llama 3
❌ Not supported: may be considered for future releases
N/A Not applicable to this (dense) architecture

Runner is the model runner that the plugin selects by default: Model Runner V2 (MRv2) or Model Runner V1 (MRv1).

The deployment tutorial walks through deploying Llama 3.3 70B in static FP8 on a single Trn2 or Trn3 instance with tensor parallelism.

Decode context-length bucketing and component data parallelism also select MRv1. See Model Runner V2 for details.

Known issues#

Data parallelism (DP) throughput does not scale as expected, due to a vLLM DP load-balancing issue. Mitigation: run multiple DP1 servers (e.g. 4× DP1 in place of one DP4) and let a higher-level router balance requests across them.

Accuracy Evaluation#

Accuracy measured on real hardware against the Neuron 2.32 release build (plugin sha f9548908). GSM8K scores use flexible-extract (n=100).

Model	Precision	GSM8K
Llama 3.3 70B	BF16	0.95
Llama 3.3 70B	Static FP8	0.95

FP8 is within measurement noise of BF16 on GSM8K at 70B — there is no meaningful accuracy cost from quantization at this scale.

Reproduce: Serve the checkpoint following the tutorial, then run the evaluation harness against the running server over its OpenAI-compatible endpoint. See the accuracy debugging guide for the evaluation workflow.

Tutorials#

Tutorial: Deploy Llama 3.3 70B (FP8) with vLLM Neuron — End-to-end single-instance deployment recipe for Llama 3.3 70B in static FP8.
EAGLE3 speculative decoding tutorial — Add an EAGLE3 draft model for higher decode throughput.