This document is relevant for: Inf1, Inf2, Trn1, Trn1n

Inference

Note

For help selecting a framework type, see:

Comparison of torch-neuron (Inf1) versus torch-neuronx (Inf2 & Trn1) for Inference

Inference on Inf2 & Trn1 (torch-neuronx)

Tutorials (torch-neuronx)

• HuggingFace pretrained BERT tutorial [html] [notebook]

• TorchServe tutorial [html]

• LibTorch C++ tutorial (for torch-neuron and torch-neuronx) [html]

• Torchvision ResNet50 tutorial [html] [notebook]

Note

To use Jupyter Notebook see:

• Jupyter Notebook QuickStart

• Running Jupyter Notebook as script

Additional Examples (torch-neuronx)

• AWS Neuron Samples GitHub Repository

• Transformers Neuron GitHub samples

API Reference Guide (torch-neuronx)

• PyTorch Neuron (torch-neuronx) Tracing API for Inference

• PyTorch Neuron (torch-neuronx) NeuronCore Placement APIs [Experimental]

• PyTorch Neuron (torch-neuronx) Analyze API for Inference

Developer Guide (torch-neuronx)

• NeuronCore Allocation and Model Placement for Inference (torch-neuronx)

• Comparison of Traced Inference versus XLA Lazy Tensor Inference (torch-neuronx)

Misc (torch-neuronx)

• PyTorch Neuron (torch-neuronx) release notes

Inference on Inf1 (torch-neuron)

Tutorials (torch-neuron)

Computer Vision Tutorials

• ResNet-50 tutorial [html] [notebook]

• PyTorch YOLOv4 tutorial [html] [notebook]

Natural Language Processing (NLP) Tutorials

• HuggingFace pretrained BERT tutorial [html] [notebook]

• Bring your own HuggingFace pretrained BERT container to Sagemaker Tutorial [html] [notebook]

• LibTorch C++ tutorial [html]

• TorchServe tutorial [html]

• HuggingFace MarianMT tutorial [html] [notebook]

Utilizing Neuron Capabilities Tutorials

• BERT TorchServe tutorial [html]

• NeuronCore Pipeline tutorial [html] [notebook]

Note

To use Jupyter Notebook see:

• Jupyter Notebook QuickStart

• Running Jupyter Notebook as script

Additional Examples (torch-neuron)

• AWS Neuron Samples GitHub Repository

API Reference Guide (torch-neuron)

• PyTorch Neuron trace Python API

• torch.neuron.DataParallel API

• PyTorch Neuron (torch-neuron) Core Placement API [Experimental]

Developer Guide (torch-neuron)

• Running Inference on Variable Input Shapes with Bucketing

• Data Parallel Inference on PyTorch Neuron

• Developer Guide - PyTorch Neuron (torch-neuron) LSTM Support

• PyTorch Neuron (torch-neuron) Core Placement

Misc (torch-neuron)

• PyTorch Neuron (torch-neuron) Supported operators

• Troubleshooting Guide for PyTorch Neuron (torch-neuron)

• PyTorch Neuron (torch-neuron) release notes

This document is relevant for: Inf1, Inf2, Trn1, Trn1n