This document is relevant for: Trn2, Trn3

Llama 3 Model Recipe#

Introduction#

Llama 3 is a family of open-weight, decoder-only text language models from Meta, instruction-tuned for chat and general task-following. This recipe covers three sizes across the Llama 3.1 / 3.2 / 3.3 releases — a 1B, an 8B, and a 70B — which share the same architecture and serving path and differ only in scale and the parallelism degree used to serve them.

Llama 3 is supported for inference serving with vLLM using the Neuron SDK on AWS Trainium2 (trn2) and Trainium3 (trn3) hardware.

Compatible model checkpoints:

Model	HuggingFace	Hardware	Quantization
Llama 3.2 1B Instruct	meta-llama/Llama-3.2-1B-Instruct	Trn2, Trn3	BF16, FP8 (static)
Llama 3.1 8B Instruct	meta-llama/Llama-3.1-8B-Instruct	Trn2, Trn3	BF16, FP8 (static)
Llama 3.3 70B Instruct	meta-llama/Llama-3.3-70B-Instruct	Trn2, Trn3	BF16, FP8 (static)

The three sizes are byte-identical in serving path; only the tensor-parallel degree changes. All three run in BF16 out of the box, and all three accept a static FP8 checkpoint (see below). There is no quantization difference between Trn2 and Trn3 — both run the same static FP8 checkpoints.

Quantization#

Llama 3 supports tensor-wide (per-tensor) static FP8 checkpoints in the NVIDIA ModelOpt format, for example nvidia/Llama-3.3-70B-Instruct-FP8. The checkpoint carries per-tensor static weight, activation, and KV-cache scales in its quantization_config (quant_method: "modelopt", quant_algo: "FP8"). The Neuron backend detects the format and loads the FP8 weights automatically — you do not set vLLM’s --quantization flag. See the FP8 static weight quantization section of the features guide for details.

Both Trn2 and Trn3 run the same static FP8 checkpoints. Internally, Trn3 routes FP8 matmuls through STATIC_MX kernels for a prefill speedup, but the checkpoint and the way you launch the server are identical on both platforms.

Speculative decoding#

Llama 3 supports EAGLE3 speculative decoding. Pair a target checkpoint with a matching EAGLE3 draft, for example RedHatAI/Llama-3.3-70B-Instruct-speculator.eagle3 as the draft for meta-llama/Llama-3.3-70B-Instruct. Under greedy sampling EAGLE3 is a lossless acceleration — the target model’s output is unchanged. See the EAGLE3 speculative decoding tutorial for an end-to-end walkthrough and acceptance-rate tuning.

Features#

Per-model feature availability for Llama 3. See the features guide for configuration details and the cross-model feature compatibility matrix.

Category	Feature	Status
Inputs	Text	✅
Quantization	BF16 weights	✅
	FP8 static (per-tensor, ModelOpt)	✅
	KV cache FP8	✅
Parallelism	Tensor parallelism (TP)	✅
	Data parallelism (DP)	✅ (see Known issues)
	Expert parallelism (EP)	N/A
	Pipeline parallelism (PP)	❌
Performance	Continuous batching	✅
	Segmented prefill	✅
	Prefix caching (APC)	✅
	Speculative decoding (EAGLE3)	✅
	Disaggregated inference (1P1D / xPyD)	✅
	On-device sampling (greedy, top-k, top-p)	✅
Serving	Structured outputs / tool calling	✅
Compilation	torch.compile (XLA backend)	✅
	CPU mode (testing)	✅

Status legend:

✅ Supported: integrated and tested for Llama 3
❌ Not supported: may be considered for future releases
N/A Not applicable to this (dense) architecture

The deployment tutorial walks through deploying Llama 3.3 70B in static FP8 on a single Trn2 or Trn3 instance with tensor parallelism.

Known issues#

Data parallelism (DP) throughput does not scale as expected, due to a vLLM DP load-balancing issue. Mitigation: run multiple DP1 servers (e.g. 4× DP1 in place of one DP4) and let a higher-level router balance requests across them.

Accuracy Evaluation#

Accuracy measured on real hardware against the Neuron 2.32 release build (plugin sha f9548908). GSM8K scores use flexible-extract (n=100).

Model	Precision	GSM8K
Llama 3.3 70B	BF16	0.95
Llama 3.3 70B	Static FP8	0.95

FP8 is within measurement noise of BF16 on GSM8K at 70B — there is no meaningful accuracy cost from quantization at this scale.

Reproduce: Serve the checkpoint following the tutorial, then run the evaluation harness against the running server over its OpenAI-compatible endpoint. See the accuracy debugging guide for the evaluation workflow.

Tutorials#

Tutorial: Deploy Llama 3.3 70B (FP8) with vLLM Neuron — End-to-end single-instance deployment recipe for Llama 3.3 70B in static FP8.
EAGLE3 speculative decoding tutorial — Add an EAGLE3 draft model for higher decode throughput.