This document is relevant for: Inf1, Trn1

PyTorch Neuron

PyTorch Neuron unlocks high-performance and cost-effective deep learning acceleration on AWS Trainium-based and Inferentia-based Amazon EC2 instances.

PyTorch Neuron plugin architecture enables native PyTorch models to be accelerated on Neuron devices, so you can use your existing framework application and get started easily with minimal code changes.

Training

Setup Guide for Trn1

• Fresh install
• Update to latest release
• Install previous releases

Important

AutoScalingGroups is currently not supported on Trn1 and will be added soon.

To launch a Trn1 cluster you can use AWS ParallelCluster, please see example.

Note

Neuron Driver installed on Deep Learning AMI (DLAMI) with Conda does not support Trn1.

If you want to use DLAMI with Conda, please make sure to uninstall aws-neuron-dkms and install aws-neuronx-dkms before using Neuron on DLAMI with Conda.

Tutorials (torch-neuronx)

• Hugging Face BERT Pretraining Tutorial
• Multi-Layer Perceptron Training Tutorial
• PyTorch Neuron for Trainium Hugging Face BERT MRPC task finetuning using Hugging Face Trainer API
• Megatron-LM GPT Pretraining Tutorial

Note

To use Jupyter Notebook see:

• Jupyter Notebook QuickStart

• Running Jupyter Notebook as script

Additional Examples (torch-neuronx)

• AWS Neuron Reference for Megatron-LM GitHub Repository

• AWS Neuron Samples for AWS ParallelCluster

• AWS Neuron Samples GitHub Repository

API Reference Guide (torch-neuronx)

• PyTorch Neuron neuron_parallel_compile CLI
• PyTorch Neuron Environment Variable
• PyTorch Neuron Profiling API

Developer Guide (torch-neuronx)

• Developer Guide for Training with PyTorch Neuron (torch-neuronx)
• How to debug models in PyTorch Neuron (torch-neuronx)
• Developer Guide for Profiling with PyTorch Neuron (torch-neuronx)

• PyTorch Neuron (torch-neuronx) - Training Supported Operators
• PyTorch Neuron (torch-neuronx) for Training Troubleshooting Guide
• PyTorch Neuron (torch-neuronx) release notes

Inference

Setup Guide for Inf1

• Fresh install
• Update to latest release
• Install previous releases
• Install with support for cxx11 ABI

Tutorials (torch-neuron)

Computer Vision Tutorials

• ResNet-50 tutorial [html] [notebook]

• PyTorch YOLOv4 tutorial [html] [notebook]

Natural Language Processing (NLP) Tutorials

• HuggingFace pretrained BERT tutorial [html] [notebook]

• Bring your own HuggingFace pretrained BERT container to Sagemaker Tutorial [html] [notebook]

• LibTorch C++ tutorial [html]

• HuggingFace MarianMT tutorial [html] [notebook]

Utilizing Neuron Capabilities Tutorials

• BERT TorchServe tutorial [html]

• NeuronCore Pipeline tutorial [html] [notebook]

Note

To use Jupyter Notebook see:

• Jupyter Notebook QuickStart

• Running Jupyter Notebook as script

Additional Examples (torch-neuron)

• AWS Neuron Samples GitHub Repository

API Reference Guide (torch-neuron)

• PyTorch Neuron trace Python API
• torch.neuron.DataParallel API
• Experimental: NeuronCore Placement APIs

Developer Guide (torch-neuron)

• Running Inference on Variable Input Shapes with Bucketing
• Data Parallel Inference on PyTorch Neuron
• Developer Guide - PyTorch Neuron (torch-neuron) LSTM Support
• Developer Guide - PyTorch Neuron (torch-neuron) Core Placement

• PyTorch Neuron (torch-neuron) Supported operators
• Troubleshooting Guide for PyTorch Neuron (torch-neuron)
• PyTorch Neuron (torch-neuron) release notes

This document is relevant for: Inf1, Trn1