.. _neuron_containers:

Deploy Containers with Neuron
=============================

In this section you will find resources to help you use containers for your accelerated deep learning model acceleration on top of Inferentia and Trainium enabled instances. 

The section is organized based on the target deployment environment
and use case.  In most cases, it is recommended to use a preconfigured
`Deep Learning Container (DLC) <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/what-is-dlc.html>`_ from AWS.
Each DLC is pre-configured to have all of the Neuron components installed and is specific to the chosen ML Framework.

.. dropdown::  Locate Neuron DLC image
         :class-title: sphinx-design-class-title-med
         :class-body: sphinx-design-class-body-small
         :animate: fade-in

         .. tab-set:: 

               .. tab-item:: Latest Neuron DLC images


                  Find latest `Neuron DLC images <https://github.com/aws/deep-learning-containers/blob/master/available_images.md#user-content-neuron-containers>`_.

         .. tab-set:: 

               .. tab-item:: Locate specific Neuron DLC release

                  In the `DLC release page <https://github.com/aws/deep-learning-containers/releases>`_ do a search for Neuron to get the ECR repo location of specific Neuron DLC release.


.. dropdown::  Getting Started
         :class-title: sphinx-design-class-title-med
         :class-body: sphinx-design-class-body-small
         :animate: fade-in

         .. tab-set::

            .. tab-item:: Training


               .. dropdown::  Launch Trn1 Instance
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. include:: /general/setup/install-templates/launch-trn1.txt

               .. dropdown:: Install Drivers
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. code:: bash


                        # Configure Linux for Neuron repository updates

                        sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <<EOF
                        [neuron]
                        name=Neuron YUM Repository
                        baseurl=https://yum.repos.neuron.amazonaws.com
                        enabled=1
                        metadata_expire=0
                        EOF
                        sudo rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB

                        # Update OS packages
                        sudo yum update -y

                        # Install OS headers
                        sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

                        # Remove preinstalled packages and Install Neuron Driver and Runtime
                        sudo yum remove aws-neuron-dkms -y
                        sudo yum remove aws-neuronx-dkms -y
                        sudo yum install aws-neuronx-dkms-2.*  -y

                        # Install EFA Driver(only required for multiinstance training)
                        curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz
                        wget https://efa-installer.amazonaws.com/aws-efa-installer.key && gpg --import aws-efa-installer.key
                        cat aws-efa-installer.key | gpg --fingerprint
                        wget https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz.sig && gpg --verify ./aws-efa-installer-latest.tar.gz.sig
                        tar -xvf aws-efa-installer-latest.tar.gz
                        cd aws-efa-installer && sudo bash efa_installer.sh --yes
                        cd
                        sudo rm -rf aws-efa-installer-latest.tar.gz aws-efa-installer

               .. dropdown:: Install Docker
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. code:: bash

                        sudo yum install -y docker.io
                        sudo usermod -aG docker $USER

                        Logout and log back in to refresh membership.

               .. dropdown:: Verify Docker
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. code:: bash

                        docker run hello-world

                     Expected result:

                     ::

                        Hello from Docker!
                        This message shows that your installation appears to be working correctly.

                        To generate this message, Docker took the following steps:
                        1. The Docker client contacted the Docker daemon.
                        2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
                        (amd64)
                        3. The Docker daemon created a new container from that image which runs the
                        executable that produces the output you are currently reading.
                        4. The Docker daemon streamed that output to the Docker client, which sent it
                        to your terminal.

                        To try something more ambitious, you can run an Ubuntu container with:
                        $ docker run -it ubuntu bash

                        Share images, automate workflows, and more with a free Docker ID:
                        https://hub.docker.com/

                        For more examples and ideas, visit:
                        https://docs.docker.com/get-started/

               .. dropdown:: Verify Neuron Component
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     Once the environment is setup, a container can be started with
                     --device=/dev/neuron# to specify desired set of Inferentia/Trainium devices to be
                     exposed to the container. To find out the available neuron devices on
                     your instance, use the command ``ls /dev/neuron*``.

                     When running neuron-ls inside a container, you will only see the set of
                     exposed Trainiums. For example:

                     .. code:: bash

                        docker run --device=/dev/neuron0 neuron-test neuron-ls

                        Would produce the following output in trn1.32xlarge:

                        ::

                           +--------+--------+--------+---------+
                           | NEURON | NEURON | NEURON |   PCI   |
                           | DEVICE | CORES  | MEMORY |   BDF   |
                           +--------+--------+--------+---------+
                           | 0      | 2      | 32 GB  | 10:1c.0 |
                           +--------+--------+--------+---------+

               .. dropdown:: Build and Run Docker Image
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     * :ref:`how-to-build-neuron-container`

               .. dropdown::  Run Tutorial
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     :ref:`tutorial-training`


            .. tab-item:: Inference


               .. dropdown::  Launch Inf1 Instance
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. include:: /general/setup/install-templates/launch-inf1.txt

               .. dropdown:: Install Drivers
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. code:: bash

                        # Configure Linux for Neuron repository updates
                        sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <<EOF
                        [neuron]
                        name=Neuron YUM Repository
                        baseurl=https://yum.repos.neuron.amazonaws.com
                        enabled=1
                        metadata_expire=0
                        EOF
                        sudo rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB

                        # Update OS packages
                        sudo yum update -y

                        ################################################################################################################
                        # To install or update to Neuron versions 1.19.1 and newer from previous releases:
                        # - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
                        ################################################################################################################

                        # Install OS headers
                        sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

                        # Install Neuron Driver
                        sudo yum install aws-neuron-dkms -y

                        ####################################################################################
                        # Warning: If Linux kernel is updated as a result of OS package update
                        #          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
                        ####################################################################################

               .. dropdown:: Install Docker
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. code:: bash

                        sudo yum install -y docker.io
                        sudo usermod -aG docker $USER

                        Logout and log back in to refresh membership.

               .. dropdown:: Verify Docker
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     .. code:: bash

                        docker run hello-world

                     Expected result:

                     ::

                        Hello from Docker!
                        This message shows that your installation appears to be working correctly.

                        To generate this message, Docker took the following steps:
                        1. The Docker client contacted the Docker daemon.
                        2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
                        (amd64)
                        3. The Docker daemon created a new container from that image which runs the
                        executable that produces the output you are currently reading.
                        4. The Docker daemon streamed that output to the Docker client, which sent it
                        to your terminal.

                        To try something more ambitious, you can run an Ubuntu container with:
                        $ docker run -it ubuntu bash

                        Share images, automate workflows, and more with a free Docker ID:
                        https://hub.docker.com/

                        For more examples and ideas, visit:
                        https://docs.docker.com/get-started/


               .. dropdown:: Verify Neuron Component
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     Once the environment is setup, a container can be started with
                     --device=/dev/neuron# to specify desired set of Inferentia/Trainium devices to be
                     exposed to the container. To find out the available neuron devices on
                     your instance, use the command ``ls /dev/neuron*``.

                     When running neuron-ls inside a container, you will only see the set of
                     exposed Inferentias. For example:

                     .. code:: bash

                        docker run --device=/dev/neuron0 neuron-test neuron-ls

                        Would produce the following output in inf1.xlarge:

                        ::

                           +--------------+---------+--------+-----------+-----------+------+------+
                           |   PCI BDF    | LOGICAL | NEURON |  MEMORY   |  MEMORY   | EAST | WEST |
                           |              |   ID    | CORES  | CHANNEL 0 | CHANNEL 1 |      |      |
                           +--------------+---------+--------+-----------+-----------+------+------+
                           | 0000:00:1f.0 |       0 |      4 | 4096 MB   | 4096 MB   |    0 |    0 |
                           +--------------+---------+--------+-----------+-----------+------+------+

               .. dropdown::  Run Tutorial
                     :class-title: sphinx-design-class-title-small
                     :class-body: sphinx-design-class-body-small
                     :animate: fade-in

                     :ref:`tutorial-infer`



.. dropdown::  Kubernetes Getting Started
         :class-title: sphinx-design-class-title-med
         :class-body: sphinx-design-class-body-small
         :animate: fade-in

         .. dropdown:: Prerequisite
               :class-title: sphinx-design-class-title-small
               :class-body: sphinx-design-class-body-small
               :animate: fade-in

               .. include:: /containers/tutorials/k8s-prerequisite.rst

         .. dropdown:: Deploy Neuron Device Plugin
               :class-title: sphinx-design-class-title-small
               :class-body: sphinx-design-class-body-small
               :animate: fade-in

               .. include:: /containers/tutorials/k8s-neuron-device-plugin.rst

         .. dropdown:: Deploy Neuron Scheduler Extension
               :class-title: sphinx-design-class-title-small
               :class-body: sphinx-design-class-body-small
               :animate: fade-in

               .. include:: /containers/tutorials/k8s-neuron-scheduler.rst


.. dropdown::  Tutorials
        :class-title: sphinx-design-class-title-med
        :class-body: sphinx-design-class-body-small
        :animate: fade-in

        .. tab-set:: 

            .. tab-item:: Training

               .. toctree:: 
                     :maxdepth: 1

                     tutorials/training/tutorial-training
                     tutorials/training/k8s_mlp_train_demo

        .. tab-set:: 

            .. tab-item:: Inference

               .. toctree:: 
                     :maxdepth: 1

                     tutorials/inference/tutorial-infer
                     tutorials/inference/k8s_rn50_demo


.. dropdown::  Developer Flows
        :class-title: sphinx-design-class-title-med
        :class-body: sphinx-design-class-body-small
        :animate: fade-in

        .. tab-set:: 

            .. tab-item:: Inference

               .. toctree:: 
                     :maxdepth: 1


                     dlc-then-ec2-devflow
                     dlc-then-ecs-devflow
                     dlc-then-eks-devflow
                     container-sm-hosting-devflow

.. dropdown::  FAQ, Troubleshooting and Release Note
         :class-title: sphinx-design-class-title-med
         :class-body: sphinx-design-class-body-small
         :animate: fade-in
         :open:

         .. toctree::
            :maxdepth: 1

            FAQ </containers/faq>
            troubleshooting
            /release-notes/containers/neuron-containers
            /release-notes/containers/neuron-k8