This document is relevant for: Inf1

Run inference in pytorch neuron container#

Table of Contents


This tutorial demonstrates how to run a pytorch DLC on an inferentia instance.

By the end of this tutorial you will be able to run the inference using the container

You will use an inf1.2xlarge to test your Docker configuration for Inferentia.

To find out the available neuron devices on your instance, use the command ls /dev/neuron*.

Setup Environment#

  1. Launch an Inf1 Instance
    • Please follow the instructions at launch an Amazon EC2 Instance to Launch an Inf1 instance, when choosing the instance type at the EC2 console. Please make sure to select the correct instance type. To get more information about Inf1 instances sizes and pricing see Inf1 web page.

    • Select your Amazon Machine Image (AMI) of choice, please note that Neuron support Amazon Linux 2 AMI(HVM) - Kernel 5.10

    • After launching the instance, follow the instructions in Connect to your instance to connect to the instance

  2. Set up docker environment according to Tutorial Docker environment setup

3. A sample Dockerfile for for torch-neuron can be found here DLC sample Dockerfile for Application Container. This dockerfile needs the torchserve entrypoint found here Torchserve Example and torchserve found here Torchserve example.

With the files in a dir, build the image with the following command:

docker build . -f -t neuron-container:pytorch

Run the following command to start the container

docker run -itd --name pt-cont -p 80:8080 -p 8081:8081 --device=/dev/neuron0 neuron-container:pytorch /usr/local/bin/ -m 'pytorch-resnet-neuron=' -t /home/model-server/

This document is relevant for: Inf1