This document is relevant for: Inf2, Trn1, Trn1n

Inference Samples/Tutorials (Inf2/Trn1)

Encoders

Model	Frameworks/Libraries	Samples and Tutorials
bert-base-cased-finetuned-mrpc	torch-neuronx	BERT TorchServe tutorial HuggingFace pretrained BERT tutorial [html] [notebook] LibTorch C++ Tutorial for HuggingFace Pretrained BERT Compiling and Deploying HuggingFace Pretrained BERT on Inf2 on Amazon SageMaker
bert-base-cased-finetuned-mrpc	neuronx-distributed	Inference with Tensor Parallelism (neuronx-distributed) [Beta]
bert-base-uncased	torch-neuronx	HuggingFace Pretrained BERT Inference on Trn1
distilbert-base-uncased	torch-neuronx	HuggingFace Pretrained DistilBERT Inference on Trn1
roberta-base	tensorflow-neuronx	HuggingFace Roberta-Base [html] [notebook]
roberta-large	torch-neuronx	HuggingFace Pretrained RoBERTa Inference on Trn1

Decoders

Model	Frameworks/Libraries	Samples and Tutorials
gpt2	torch-neuronx	HuggingFace Pretrained GPT2 Feature Extraction on Trn1
meta-llama/Llama-2-13b	transformers-neuronx	Run Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1
meta-llama/Llama-2-70b	transformers-neuronx	Run Hugging Face meta-llama/Llama-2-70b autoregressive sampling on Inf2 & Trn1 Run speculative sampling on Meta Llama models [Beta]
meta-llama/Llama-2-7b	neuronx-distributed	Run Hugging Face meta-llama/Llama-2-7b autoregressive sampling on Inf2 & Trn1 ([html] [notebook])
mistralai/Mistral-7B-Instruct-v0.1	transformers-neuronx	Run Mistral-7B-Instruct-v0.1 autoregressive sampling on Inf2 & Trn1
mistralai/Mistral-7B-Instruct-v0.2	transformers-neuronx	Run Hugging Face mistralai/Mistral-7B-Instruct-v0.2 autoregressive sampling on Inf2 & Trn1 [Beta]
Mixtral-8x7B-v0.1	transformers-neuronx	Run Hugging Face mistralai/Mixtral-8x7B-v0.1 autoregressive sampling on Inf2 & Trn1
codellama/CodeLlama-13b-hf	transformers-neuronx	Run Hugging Face codellama/CodeLlama-13b-hf autoregressive sampling on Inf2 & Trn1

Encoder-Decoders

Model	Frameworks/Libraries	Samples and Tutorials
t5-large	torch-neuronx optimum-neuron	T5 inference tutorial [html] [notebook]
t5-3b	neuronx-distributed	T5 inference tutorial [html] [notebook]
google/flan-t5-xl	neuronx-distributed	flan-t5-xl inference tutorial [html] [notebook]

Vision Transformers

Model	Frameworks/Libraries	Samples and Tutorials
google/vit-base-patch16-224	torch-neuronx	HuggingFace Pretrained ViT Inference on Trn1
clip-vit-base-patch32	torch-neuronx	HuggingFace Pretrained CLIP Base Inference on Inf2
clip-vit-large-patch14	torch-neuronx	HuggingFace Pretrained CLIP Large Inference on Inf2

Convolutional Neural Networks(CNN)

Model	Frameworks/Libraries	Samples and Tutorials
resnet50	torch-neuronx	Torchvision Pretrained ResNet50 Inference on Trn1 / Inf2 Torchvision ResNet50 tutorial [html] [notebook]
resnet50	tensorflow-neuronx	Using NEURON_RT_VISIBLE_CORES with TensorFlow Serving
unet	torch-neuronx	Pretrained UNet Inference on Trn1 / Inf2
vgg	torch-neuronx	Torchvision Pretrained VGG Inference on Trn1 / Inf2

Stable Diffusion

Model	Frameworks/Libraries	Samples and Tutorials
stable-diffusion-v1-5	torch-neuronx	HuggingFace Stable Diffusion 1.5 (512x512) Inference on Trn1 / Inf2
stable-diffusion-2-1-base	torch-neuronx	HuggingFace Stable Diffusion 2.1 (512x512) Inference on Trn1 / Inf2
stable-diffusion-2-1	torch-neuronx	HuggingFace Stable Diffusion 2.1 (768x768) Inference on Trn1 / Inf2 Deploy & Run Stable Diffusion on SageMaker and Inferentia2
stable-diffusion-xl-base-1.0	torch-neuronx	HuggingFace Stable Diffusion XL 1.0 (1024x1024) Inference on Inf2 HuggingFace Stable Diffusion XL 1.0 Base and Refiner (1024x1024) Inference on Inf2
stable-diffusion-2-inpainting	torch-neuronx	stable-diffusion-2-inpainting model Inference on Trn1 / Inf2

Multi Modal

Model	Frameworks/Libraries	Samples and Tutorials
multimodal-perceiver	torch-neuronx	HuggingFace Multimodal Perceiver Inference on Trn1 / Inf2
language-perceiver	torch-neuronx	HF Pretrained Perceiver Language Inference on Trn1 / Inf2
vision-perceiver-conv	torch-neuronx	HF Pretrained Perceiver Image Classification Inference on Trn1 / Inf2

This document is relevant for: Inf2, Trn1, Trn1n