.. _containers-getting-started:

Getting started with Neuron DLC using Docker
============================================

.. tab-set::

   .. tab-item:: Training


      .. dropdown::  Launch Trn1 Instance
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. include:: /setup/install-templates/launch-instance.txt

      .. dropdown:: Install Drivers
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. code:: bash


               # Configure Linux for Neuron repository updates

               sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <<EOF
               [neuron]
               name=Neuron YUM Repository
               baseurl=https://yum.repos.neuron.amazonaws.com
               enabled=1
               metadata_expire=0
               EOF
               sudo rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB

               # Update OS packages
               sudo dnf update -y

               # Install OS headers
               sudo dnf install -y "kernel-devel-uname-r = $(uname -r)"

               # Remove preinstalled packages and Install Neuron Driver and Runtime
               sudo dnf remove aws-neuron-dkms -y
               sudo dnf remove aws-neuronx-dkms -y
               sudo dnf install aws-neuronx-dkms-2.*  -y

               # Install EFA Driver(only required for multi-instance training)
               curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz
               wget https://efa-installer.amazonaws.com/aws-efa-installer.key && gpg --import aws-efa-installer.key
               cat aws-efa-installer.key | gpg --fingerprint
               wget https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz.sig && gpg --verify ./aws-efa-installer-latest.tar.gz.sig
               tar -xvf aws-efa-installer-latest.tar.gz
               cd aws-efa-installer && sudo bash efa_installer.sh --yes
               cd
               sudo rm -rf aws-efa-installer-latest.tar.gz aws-efa-installer

      .. dropdown:: Install Docker
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. code:: bash

               sudo dnf install -y docker.io
               sudo usermod -aG docker $USER

            Logout and log back in to refresh membership.

      .. dropdown:: Verify Docker
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. code:: bash

               docker run hello-world

            Expected result:

            ::

               Hello from Docker!
               This message shows that your installation appears to be working correctly.

               To generate this message, Docker took the following steps:
               1. The Docker client contacted the Docker daemon.
               2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
               (amd64)
               3. The Docker daemon created a new container from that image which runs the
               executable that produces the output you are currently reading.
               4. The Docker daemon streamed that output to the Docker client, which sent it
               to your terminal.

               To try something more ambitious, you can run an Ubuntu container with:
               $ docker run -it ubuntu bash

               Share images, automate workflows, and more with a free Docker ID:
               https://hub.docker.com/

               For more examples and ideas, visit:
               https://docs.docker.com/get-started/

      .. dropdown:: Verify Neuron Component
           :class-title: sphinx-design-class-title-small
           :class-body: sphinx-design-class-body-small
           :animate: fade-in

           Once the environment is setup, a container can be started with
           --device=/dev/neuron# to specify desired set of Inferentia/Trainium devices to be
           exposed to the container. To find out the available neuron devices on
           your instance, use the command ``ls /dev/neuron*``.

           When running neuron-ls inside a container, you will only see the set of
           exposed Trainiums. For example:

           .. code:: bash

             docker run --device=/dev/neuron0 neuron-test neuron-ls

           Would produce the following output in trn1.32xlarge:

           ::

             +--------+--------+--------+---------+
             | NEURON | NEURON | NEURON |   PCI   |
             | DEVICE | CORES  | MEMORY |   BDF   |
             +--------+--------+--------+---------+
             | 0      | 2      | 32 GB  | 10:1c.0 |
             +--------+--------+--------+---------+

      .. dropdown::  Run Tutorial
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            :ref:`tutorial-training`


   .. tab-item:: Inference


      .. dropdown::  Launch Inf1 Instance
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. include:: /setup/install-templates/launch-inf1.txt

      .. dropdown:: Install Drivers
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. code:: bash

               # Configure Linux for Neuron repository updates
               sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <<EOF
               [neuron]
               name=Neuron YUM Repository
               baseurl=https://yum.repos.neuron.amazonaws.com
               enabled=1
               metadata_expire=0
               EOF
               sudo rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB

               # Update OS packages
               sudo dnf update -y

               ################################################################################################################
               # To install or update to Neuron versions 1.19.1 and newer from previous releases:
               # - DO NOT skip 'aws-neuron-dkms' install or upgrade step, you MUST install or upgrade to latest Neuron driver
               ################################################################################################################

               # Install OS headers
               sudo dnf install -y "kernel-devel-uname-r = $(uname -r)"

               # Install Neuron Driver
               sudo dnf install aws-neuron-dkms -y

               ####################################################################################
               # Warning: If Linux kernel is updated as a result of OS package update
               #          Neuron driver (aws-neuron-dkms) should be re-installed after reboot
               ####################################################################################

      .. dropdown:: Install Docker
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. code:: bash

               sudo dnf install -y docker.io
               sudo usermod -aG docker $USER

            Logout and log back in to refresh membership.

      .. dropdown:: Verify Docker
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            .. code:: bash

               docker run hello-world

            Expected result:

            ::

               Hello from Docker!
               This message shows that your installation appears to be working correctly.

               To generate this message, Docker took the following steps:
               1. The Docker client contacted the Docker daemon.
               2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
               (amd64)
               3. The Docker daemon created a new container from that image which runs the
               executable that produces the output you are currently reading.
               4. The Docker daemon streamed that output to the Docker client, which sent it
               to your terminal.

               To try something more ambitious, you can run an Ubuntu container with:
               $ docker run -it ubuntu bash

               Share images, automate workflows, and more with a free Docker ID:
               https://hub.docker.com/

               For more examples and ideas, visit:
               https://docs.docker.com/get-started/


      .. dropdown:: Verify Neuron Component
           :class-title: sphinx-design-class-title-small
           :class-body: sphinx-design-class-body-small
           :animate: fade-in

           Once the environment is setup, a container can be started with
           --device=/dev/neuron# to specify desired set of Inferentia/Trainium devices to be
           exposed to the container. To find out the available neuron devices on
           your instance, use the command ``ls /dev/neuron*``.

           When running neuron-ls inside a container, you will only see the set of
           exposed Inferentias. For example:

           .. code:: bash

             docker run --device=/dev/neuron0 neuron-test neuron-ls


           Would produce the following output in inf1.xlarge:

           ::

               +--------------+---------+--------+-----------+-----------+------+------+
               |   PCI BDF    | LOGICAL | NEURON |  MEMORY   |  MEMORY   | EAST | WEST |
               |              |   ID    | CORES  | CHANNEL 0 | CHANNEL 1 |      |      |
               +--------------+---------+--------+-----------+-----------+------+------+
               | 0000:00:1f.0 |       0 |      4 | 4096 MB   | 4096 MB   |    0 |    0 |
               +--------------+---------+--------+-----------+-----------+------+------+

      .. dropdown::  Run Tutorial
            :class-title: sphinx-design-class-title-small
            :class-body: sphinx-design-class-body-small
            :animate: fade-in

            :ref:`tutorial-infer`
            :ref:`quickstart_vllm_dlc_deploy`