This document is relevant for: Inf2, Trn1, Trn1n

Neuron Calculator

Select the Calculator

NeuronCores needed for LLM Inference

Number of NeuronCores needed for LLM Inference

Please enter model configurations (You can enter multiple values of each hyperparameter. Press enter after adding each value in the text field)

Model:

Custom LLM Model opt-66b meta-llama/Llama-2-7b meta-llama/Llama-2-13b

Instance Type:

Inf2 Trn1

Data Type:

BF16 / FP16

Batch Size:

Max Sequence Length:

Embedding Dimension:

Number of Attention Heads:

Number of Layers:

Tensor Parallel Degree Constraint (Flexible tensor parallelism (TP) is not supported for certain models like GPT-J and GPT-NeoX in transformers-neuronx. Checking this box will flag a TP degree as invalid if the number of attention heads is not divisible by it.)

Invalid model configurations entered. Each text field accepts multiple values. Please press Enter after adding a new value to the text field.

Submit

Batch Size	Max Seq Length	Embedding Dimension	Num Attention Heads	Num Layers	Memory Footprint (GB)	TP Degree(NeuronCores)	Instances Recommended

Edit Model Configuration Reset Calculator

This document is relevant for: Inf2, Trn1, Trn1n