This document is relevant for: Inf2, Trn1

nosearch:

Neuron Calculator (archived)

Warning

This calculator is archived and is no longer maintained. It estimates the number of NeuronCores for transformers-neuronx models on Inf2 and Trn1, and transformers-neuronx is no longer supported. For current LLM serving, see vLLM Neuron.

Select the Calculator

NeuronCores needed for LLM Inference

Number of NeuronCores needed for LLM Inference

Please enter model configurations (You can enter multiple values of each hyperparameter. Press enter after adding each value in the text field)

Model:

Custom LLM Model opt-66b meta-llama/Llama-2-7b meta-llama/Llama-2-13b

Instance Type:

Inf2 Trn1

Data Type:

BF16 / FP16

Batch Size:

Max Sequence Length:

Embedding Dimension:

Number of Attention Heads:

Number of Layers:

Tensor Parallel Degree Constraint (Flexible tensor parallelism (TP) is not supported for certain models like GPT-J and GPT-NeoX in transformers-neuronx. Checking this box will flag a TP degree as invalid if the number of attention heads is not divisible by it.)

Invalid model configurations entered. Each text field accepts multiple values. Please press Enter after adding a new value to the text field.

Submit

Batch Size	Max Seq Length	Embedding Dimension	Num Attention Heads	Num Layers	Memory Footprint (GB)	TP Degree(NeuronCores)	Instances Recommended

Edit Model Configuration Reset Calculator

This document is relevant for: Inf2, Trn1