This document is relevant for: Inf1

Compile with Framework API and Deploy on EC2 Inf1#

Description#

Neuron developer flow on EC2

You can use a single inf1 instance as a development environment to compile and deploy Neuron models. In this developer flow, you provision an EC2 inf1 instance using a Deep Learming AMI (DLAMI) and execute the two steps of the development flow in the same instance. The DLAMI comes pre-packaged with the Neuron frameworks, compiler, and required runtimes to complete the flow. Development happens through Jupyter Notebooks or using a secure shell (ssh) connection in terminal. Follow the steps bellow to setup your environment.

Note

Model compilation can be executed on a non-inf1 instance for later deployment. Follow the same EC2 Developer Flow Setup using other instance families and leverage Amazon Simple Storage Service (S3) to share the compiled models between different instances.

Setup Environment#

1. Launch an Inf1 Instance#

  • Please follow the instructions at launch an Amazon EC2 Instance to Launch an Inf1 instance, when choosing the instance type at the EC2 console. Please make sure to select the correct instance type. To get more information about Inf1 instances sizes and pricing see Inf1 web page.

  • When choosing an Amazon Machine Image (AMI) make sure to select Deep Learning AMI with Conda Options. Please note that Neuron Conda environments are supported only in Ubuntu 18 DLAMI and Amazon Linux2 DLAMI, Neuron Conda environments are not supported in Amazon Linux DLAMI.

  • After launching the instance, follow the instructions in Connect to your instance to connect to the instance

Note

You can also launch the instance from AWS CLI, please see AWS CLI commands to launch inf1 instances.

2. Set up a development environment#

Enable PyTorch-Neuron#

Important

For successful installation or update to next releases (Neuron 1.20.0 and newer):
  • Uninstall aws-neuron-dkms by running: sudo apt remove aws-neuron-dkms or sudo yum remove aws-neuron-dkms

  • Install or upgrade to latest Neuron driver (aws-neuron-dkms) by following the “Setup Guide” instructions.

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install Neuron Driver
sudo apt-get install aws-neuronx-dkms --allow-change-held-packages -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate PyTorch
source activate

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Install Neuron Driver
sudo yum versionlock delete aws-neuronx-dkms
sudo yum install aws-neuronx-dkms -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate PyTorch
source activate

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install Neuron Driver
sudo apt-get install aws-neuronx-dkms --allow-change-held-packages -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate PyTorch
source activate aws_neuron_pytorch_p36

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Install Neuron Driver
sudo yum versionlock delete aws-neuronx-dkms
sudo yum install aws-neuronx-dkms -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate PyTorch
source activate aws_neuron_pytorch_p36

Enable TensorFlow-Neuron#

Important

For successful installation or update to next releases (Neuron 1.20.0 and newer):
  • Uninstall aws-neuron-dkms by running: sudo apt remove aws-neuron-dkms or sudo yum remove aws-neuron-dkms

  • Install or upgrade to latest Neuron driver (aws-neuron-dkms) by following the “Setup Guide” instructions.

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/awsdocs-neuron/checkouts/latest/src/helperscripts/neuronsetuphelper.py", line 1001, in <module>
    setup_cmd += nr_setup.instructions(framework=framework,action=action,framework_version=args.framework_version,os=args.os,ami=args.ami,mode=args.mode)
  File "/home/docs/checkouts/readthedocs.org/user_builds/awsdocs-neuron/checkouts/latest/src/helperscripts/neuronsetuphelper.py", line 977, in instructions
    setup_cmd=hlpr_instructions(self,self.neuron_version)
  File "/home/docs/checkouts/readthedocs.org/user_builds/awsdocs-neuron/checkouts/latest/src/helperscripts/neuronsetuphelper.py", line 787, in hlpr_instructions
    fal_supported_rtd=nr_setup.fal_supported_runtime[fw][fw_ver]['neuron-rtd']
KeyError: '2.10.1'

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/awsdocs-neuron/checkouts/latest/src/helperscripts/neuronsetuphelper.py", line 1001, in <module>
    setup_cmd += nr_setup.instructions(framework=framework,action=action,framework_version=args.framework_version,os=args.os,ami=args.ami,mode=args.mode)
  File "/home/docs/checkouts/readthedocs.org/user_builds/awsdocs-neuron/checkouts/latest/src/helperscripts/neuronsetuphelper.py", line 977, in instructions
    setup_cmd=hlpr_instructions(self,self.neuron_version)
  File "/home/docs/checkouts/readthedocs.org/user_builds/awsdocs-neuron/checkouts/latest/src/helperscripts/neuronsetuphelper.py", line 787, in hlpr_instructions
    fal_supported_rtd=nr_setup.fal_supported_runtime[fw][fw_ver]['neuron-rtd']
KeyError: '2.10.1'

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install Neuron Driver
sudo apt-get install aws-neuronx-dkms --allow-change-held-packages -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate TensorFlow
source activate aws_neuron_tensorflow_p36

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Install Neuron Driver
sudo yum versionlock delete aws-neuronx-dkms
sudo yum install aws-neuronx-dkms -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate TensorFlow
source activate aws_neuron_tensorflow_p36

Enable Apache MXNet#

Important

For successful installation or update to next releases (Neuron 1.20.0 and newer):
  • Uninstall aws-neuron-dkms by running: sudo apt remove aws-neuron-dkms or sudo yum remove aws-neuron-dkms

  • Install or upgrade to latest Neuron driver (aws-neuron-dkms) by following the “Setup Guide” instructions.

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Note: There is no DLAMI Conda environment for this framework version
#       Framework will be installed/updated inside a Python environment

# Update OS packages
sudo apt-get update -y

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install Neuron Driver
sudo apt-get install aws-neuronx-dkms --allow-change-held-packages -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################

# Install Neuron Tools
sudo apt-get install aws-neuronx-tools -y

export PATH=/opt/aws/neuron/bin:$PATH

# Activate MXNet
source activate aws_neuron_mxnet_p36

# Set Pip repository  to point to the Neuron repository
pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

#Install Neuron MXNet
wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl
pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl
pip install mx_neuron neuron-cc

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Note: There is no DLAMI Conda environment for this framework version
#       Framework will be installed/updated inside a Python environment

# Update OS packages
sudo yum update -y

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Install Neuron Driver
sudo yum versionlock delete aws-neuronx-dkms
sudo yum install aws-neuronx-dkms -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################

# Install Neuron Tools
sudo yum install aws-neuronx-tools -y

export PATH=/opt/aws/neuron/bin:$PATH

# Activate MXNet
source activate aws_neuron_mxnet_p36

# Set Pip repository  to point to the Neuron repository
pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

#Install Neuron MXNet
wget https://aws-mx-pypi.s3.us-west-2.amazonaws.com/1.8.0/aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl
pip install aws_mx-1.8.0.2-py2.py3-none-manylinux2014_x86_64.whl
pip install mx_neuron neuron-cc

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install Neuron Driver
sudo apt-get install aws-neuronx-dkms --allow-change-held-packages -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate MXNet
source activate aws_neuron_mxnet_p36

Note

For a successful installation or update, execute each line of the instructions below separately or copy the contents of the code block into a script file and source its contents.

# Neuron is pre-installed on Deep Learning AMI (DLAMI), latest DLAMI version may not include latest Neuron versions 
# To update to latest Neuron version, follow "Update to latest release" instruction on Neuron documentation

################################################################################################################
# To install or update to Neuron versions 1.19.1 and newer from previous releases:
# - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
################################################################################################################

# Install OS headers
sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Install Neuron Driver
sudo yum versionlock delete aws-neuronx-dkms
sudo yum install aws-neuronx-dkms -y

####################################################################################
# Warning: If Linux kernel is updated as a result of OS package update
#          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
####################################################################################


# Activate MXNet
source activate aws_neuron_mxnet_p36

3. Set up Jupyter notebook#

To develop from a Jupyter notebook see Jupyter Notebook QuickStart

You can also run a Jupyter notebook as a script, first enable the ML framework Conda or Python environment of your choice and see Running Jupyter Notebook as script for instructions.

This document is relevant for: Inf1