This document is relevant for: Inf1

Select a Different Framework or Platform for Setup

PyTorch Neuron (“torch-neuron”) Setup on Amazon Linux 2

Note

As of 2.20.0, Neuron Runtime no longer supports AL2. Upgrade to AL2023 following the AL2 Migration guide

Note

Please make sure to upgrade from python 3.7 to python 3.8 to use Neuron SDK on Amazon Linux 2. Starting from Neuron Release 2.13, python 3.7 is no longer supported as mentioned here. Also, we do not have support for torch-neuronx 2.1.2 on Amaznon Linux 2.

Get Started with Latest Release of PyTorch Neuron (torch-neuron)

This section provide links that will assist you to quickly start with a fresh installation of PyTorch Neuron (torch-neuron) Setup for Inference.

Launch the Instance

• Please follow the instructions at launch an Amazon EC2 Instance to launch an instance. When choosing the instance type at the EC2 console, please make sure to select the correct instance type.

• To get more information about instances sizes and pricing see: Inf1 web page

• Select Amazon Linux 2 AMI(HVM) - Kernel 5.10

• After launching the instance, follow the instructions in Connect to your instance to connect to the instance

Install Drivers and Tools

# Configure Linux for Neuron repository updates
sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <<EOF
[neuron]
name=Neuron YUM Repository
baseurl=https://yum.repos.neuron.amazonaws.com
enabled=1
metadata_expire=0
EOF
sudo rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB

# Update OS packages 
sudo yum update -y

# Install OS headers 
sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Install git 
sudo yum install git -y

# install Neuron Driver
sudo yum install aws-neuronx-dkms-2.* -y

# Install Neuron Runtime 
sudo yum install aws-neuronx-collectives-.* -y
sudo yum install aws-neuronx-runtime-lib-.* -y

# Install Neuron Tools 
sudo yum install aws-neuronx-tools-2.* -y

# Add PATH
export PATH=/opt/aws/neuron/bin:$PATH

Install PyTorch Neuron (torch-neuron)

# Install Python 
sudo yum install -y amazon-linux-extras
sudo yum install python3.9

# Install Python venv 
sudo yum install -y python3.9-venv gcc-c++ 

# Create Python venv
python3.9 -m venv aws_neuron_venv_pytorch_inf1 

# Activate Python venv 
source aws_neuron_venv_pytorch_inf1/bin/activate 
python -m pip install -U pip 

# Install Jupyter notebook kernel
pip install ipykernel 
python3.9 -m ipykernel install --user --name aws_neuron_venv_pytorch_inf1 --display-name "Python (torch-neuron)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository 
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Install PyTorch Neuron
python -m pip install torch-neuron neuron-cc[tensorflow] "protobuf" torchvision

Visit PyTorch Neuron(torch-neuron) for Inference section

Visit PyTorch Neuron section for more

Update to latest PyTorch Neuron (torch-neuron)

If you already have a previous Neuron release installed, this section provide links that will assist you to update to latest Neuron release.

PyTorch 1.13.1

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Activate Python venv 
source aws_neuron_venv_pytorch_inf1/bin/activate 

# Install Jupyter notebook kernel
pip install ipykernel 
python3.9 -m ipykernel install --user --name aws_neuron_venv_pytorch_inf1 --display-name "Python (torch-neuron)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository 
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Update PyTorch Neuron
python -m pip install --upgrade torch-neuron neuron-cc[tensorflow] "protobuf" torchvision

Install Previous PyTorch Neuron Releases for Amazon Linux (torch-neuron)

This section will assist you to install previous Neuron releases.

Neuron 2.18.0

# Install Python venv 
sudo yum install -y python3.8-venv gcc-c++ 

# Create Python venv
python3.8 -m venv aws_neuron_venv_pytorch_inf1 

# Activate Python venv 
source aws_neuron_venv_pytorch_inf1/bin/activate 
python -m pip install -U pip 

# Install Jupyter notebook kernel
pip install ipykernel 
python3.8 -m ipykernel install --user --name aws_neuron_venv_pytorch_inf1 --display-name "Python (torch-neuron)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository 
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Install PyTorch Neuron
python -m pip install torch-neuron==1.13.1.2.9.74.0 neuron-cc[tensorflow] "protobuf" torchvision

Neuron 2.17.0

# Install Python venv 
sudo yum install -y python3.8-venv gcc-c++ 

# Create Python venv
python3.8 -m venv aws_neuron_venv_pytorch_inf1 

# Activate Python venv 
source aws_neuron_venv_pytorch_inf1/bin/activate 
python -m pip install -U pip 

# Install Jupyter notebook kernel
pip install ipykernel 
python3.8 -m ipykernel install --user --name aws_neuron_venv_pytorch_inf1 --display-name "Python (torch-neuron)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository 
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Install PyTorch Neuron
python -m pip install torch-neuron==1.13.1.2.9.17.0 neuron-cc[tensorflow] "protobuf" torchvision

Neuron 2.16.0

# Install Python venv 
sudo yum install -y python3.8-venv gcc-c++ 

# Create Python venv
python3.8 -m venv aws_neuron_venv_pytorch_inf1 

# Activate Python venv 
source aws_neuron_venv_pytorch_inf1/bin/activate 
python -m pip install -U pip 

# Install Jupyter notebook kernel
pip install ipykernel 
python3.8 -m ipykernel install --user --name aws_neuron_venv_pytorch_inf1 --display-name "Python (torch-neuron)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository 
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Install PyTorch Neuron
python -m pip install torch-neuron==1.13.1.2.9.17.0 neuron-cc[tensorflow] "protobuf" torchvision

This document is relevant for: Inf1