This document is relevant for: Inf2, Trn1, Trn1n

Inference Samples/Tutorials (Inf2/Trn1)#

Encoders#

Model

Frameworks/Libraries

Samples and Tutorials

bert-base-cased-finetuned-mrpc

torch-neuronx

bert-base-cased-finetuned-mrpc

neuronx-distributed

bert-base-uncased

torch-neuronx

distilbert-base-uncased

torch-neuronx

roberta-base

tensorflow-neuronx

roberta-large

torch-neuronx

Decoders#

Model

Frameworks/Libraries

Samples and Tutorials

gpt2

torch-neuronx

meta-llama/Llama-3-8b

transformers-neuronx

meta-llama/Llama-3-70b

transformers-neuronx

meta-llama/Llama-2-13b

transformers-neuronx

meta-llama/Llama-2-70b

transformers-neuronx

meta-llama/Llama-2-7b

neuronx-distributed

  • Run Hugging Face meta-llama/Llama-2-7b autoregressive sampling on Inf2 & Trn1 ([html] [notebook])

meta-llama/codellama-13b

neuronx-distributed

mistralai/Mistral-7B-Instruct-v0.1

transformers-neuronx

mistralai/Mistral-7B-Instruct-v0.2

transformers-neuronx

Mixtral-8x7B-v0.1

transformers-neuronx

codellama/CodeLlama-13b-hf

transformers-neuronx

Encoder-Decoders#

Model

Frameworks/Libraries

Samples and Tutorials

t5-large

  • torch-neuronx

  • optimum-neuron

t5-3b

neuronx-distributed

google/flan-t5-xl

neuronx-distributed

Vision Transformers#

Model

Frameworks/Libraries

Samples and Tutorials

google/vit-base-patch16-224

torch-neuronx

clip-vit-base-patch32

torch-neuronx

clip-vit-large-patch14

torch-neuronx

Convolutional Neural Networks(CNN)#

Model

Frameworks/Libraries

Samples and Tutorials

resnet50

torch-neuronx

resnet50

tensorflow-neuronx

unet

torch-neuronx

vgg

torch-neuronx

Multi Modal#

Model

Frameworks/Libraries

Samples and Tutorials

multimodal-perceiver

torch-neuronx

language-perceiver

torch-neuronx

vision-perceiver-conv

torch-neuronx

This document is relevant for: Inf2, Trn1, Trn1n