This document is relevant for: Inf1, Inf2, Trn1, Trn1n

What’s New#

Neuron 2.17.0 (02/13/2024)#

What’s New#

Neuron 2.17 release improves small collective communication operators (smaller than 16MB) by up to 30%, which improves large language model (LLM) Inference performance by up to 10%. This release also includes improvements in Neuron Profiler and other minor enhancements and bug fixes.

For more detailed release notes of the new features and resolved issues, see Neuron Components Release Notes.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see Model Architecture Fit Guidelines.

Neuron Components Release Notes#

Inf1, Trn1/Trn1n and Inf2 common packages#

Component

Instance/s

Package/s

Details

Neuron Runtime

Trn1/Trn1n, Inf1, Inf2

  • Trn1/Trn1n: aws-neuronx-runtime-lib (.deb, .rpm)

  • Inf1: Runtime is linked into the ML frameworks packages

Neuron Runtime Driver

Trn1/Trn1n, Inf1, Inf2

  • aws-neuronx-dkms (.deb, .rpm)

Neuron System Tools

Trn1/Trn1n, Inf1, Inf2

  • aws-neuronx-tools (.deb, .rpm)

Containers

Trn1/Trn1n, Inf1, Inf2

  • aws-neuronx-k8-plugin (.deb, .rpm)

  • aws-neuronx-k8-scheduler (.deb, .rpm)

  • aws-neuronx-oci-hooks (.deb, .rpm)

NeuronPerf (Inference only)

Trn1/Trn1n, Inf1, Inf2

  • neuronperf (.whl)

TensorFlow Model Server Neuron

Trn1/Trn1n, Inf1, Inf2

  • tensorflow-model-server-neuronx (.deb, .rpm)

Neuron Documentation

Trn1/Trn1n, Inf1, Inf2

Trn1/Trn1n and Inf2 only packages#

Component

Instance/s

Package/s

Details

PyTorch Neuron

Trn1/Trn1n, Inf2

  • torch-neuronx (.whl)

TensorFlow Neuron

Trn1/Trn1n, Inf2

  • tensorflow-neuronx (.whl)

Neuron Compiler (Trn1/Trn1n, Inf2 only)

Trn1/Trn1n, Inf2

  • neuronx-cc (.whl)

Collective Communication library

Trn1/Trn1n, Inf2

  • aws-neuronx-collective (.deb, .rpm)

Neuron Custom C++ Operators

Trn1/Trn1n, Inf2

  • aws-neuronx-gpsimd-customop (.deb, .rpm)

  • aws-neuronx-gpsimd-tools (.deb, .rpm)

Transformers Neuron

Trn1/Trn1n, Inf2

  • transformers-neuronx (.whl)

Neuron Distributed

Trn1/Trn1n, Inf2

  • neuronx-distributed (.whl)

AWS Neuron Reference for NeMo Megatron

Trn1/Trn1n

Note

In next releases aws-neuronx-tools and aws-neuronx-runtime-lib will add support for Inf1.

Release Artifacts#

Trn1 packages#

List of packages in Neuron 2.17.0:

Component                           Package                                           
Collective Communication Library    aws-neuronx-collectives-2.20.11.0 
Driver                              aws-neuronx-dkms-2.15.9.0 
nan                                 aws-neuronx-gpsimd-customop-lib-0.9.1.0 
CustomOps Tools                     aws-neuronx-gpsimd-tools-0.9.0.0 
Kubernetes Plugin                   aws-neuronx-k8-plugin-2.19.16.0 
Kubernetes Scheduler                aws-neuronx-k8-scheduler-2.19.16.0 
OCI                                 aws-neuronx-oci-hook-2.2.45.0 
General                             aws-neuronx-runtime-discovery-2.9 
Runtime Library                     aws-neuronx-runtime-lib-2.20.11.0 
System Tools                        aws-neuronx-tools-2.17.0.0 
Framework                           libneuronxla-2.0.755 
Framework                           libneuronxla-0.5.809 
Compiler                            neuronx-cc-2.12.68.0 
Neuron Distributed                  neuronx_distributed-0.6.0 
Compiler                            neuronx_hwm-2.12.0.0 
TensorBoard                         tensorboard-plugin-neuronx-2.6.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.10.1.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.7.4.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.8.4.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.9.3.2.10.8.0 
TensorFlow                          tensorflow-neuronx-2.10.1.2.1.0 
TensorFlow                          tensorflow-neuronx-2.7.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.8.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.9.3.2.1.0 
PyTorch                             torch-neuronx-1.13.1.1.13.1 
PyTorch                             torch-neuronx-2.0.0.2.0.1b0 
PyTorch                             torch-neuronx-2.1.1.2.0.1b0 
PyTorch                             torch_xla-1.13.1+torchneurond 
PyTorch                             torch_xla-2.1.1 
Transformers Neuron                 transformers-neuronx-0.9.474

Inf2 packages#

List of packages in Neuron 2.17.0:

Component                           Package                                           
Collective Communication Library    aws-neuronx-collectives-2.20.11.0 
Driver                              aws-neuronx-dkms-2.15.9.0 
nan                                 aws-neuronx-gpsimd-customop-lib-0.9.1.0 
CustomOps Tools                     aws-neuronx-gpsimd-tools-0.9.0.0 
Kubernetes Plugin                   aws-neuronx-k8-plugin-2.19.16.0 
Kubernetes Scheduler                aws-neuronx-k8-scheduler-2.19.16.0 
OCI                                 aws-neuronx-oci-hook-2.2.45.0 
General                             aws-neuronx-runtime-discovery-2.9 
Runtime Library                     aws-neuronx-runtime-lib-2.20.11.0 
System Tools                        aws-neuronx-tools-2.17.0.0 
Framework                           libneuronxla-2.0.755 
Framework                           libneuronxla-0.5.809 
Compiler                            neuronx-cc-2.12.68.0 
Neuron Distributed                  neuronx_distributed-0.6.0 
Compiler                            neuronx_hwm-2.12.0.0 
TensorBoard                         tensorboard-plugin-neuronx-2.6.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.10.1.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.7.4.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.8.4.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.9.3.2.10.8.0 
TensorFlow                          tensorflow-neuronx-2.10.1.2.1.0 
TensorFlow                          tensorflow-neuronx-2.8.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.9.3.2.1.0 
PyTorch                             torch-neuronx-1.13.1.1.13.1 
PyTorch                             torch-neuronx-2.0.0.2.0.1b0 
PyTorch                             torch-neuronx-2.1.1.2.0.1b0 
PyTorch                             torch_xla-1.13.1+torchneurond 
PyTorch                             torch_xla-2.1.1 
Transformers Neuron                 transformers-neuronx-0.9.474

Inf1 packages#

List of packages in Neuron 2.17.0:

Component                           Package                                           
Driver                              aws-neuronx-dkms-2.15.9.0 
Kubernetes Plugin                   aws-neuronx-k8-plugin-2.19.16.0 
Kubernetes Scheduler                aws-neuronx-k8-scheduler-2.19.16.0 
OCI                                 aws-neuronx-oci-hook-2.2.45.0 
System Tools                        aws-neuronx-tools-2.17.0.0 
Compiler                            dmlc_nnvm-1.18.2.0 
Compiler                            dmlc_topi-1.18.2.0 
Compiler                            dmlc_tvm-1.18.2.0 
Compiler                            inferentia_hwm-1.16.2.0 
MXNet                               mx_neuron-1.8.0.2.4.40.0 
MXNet                               mxnet_neuron-1.5.1.1.10.0.0 
Compiler                            neuron-cc-1.21.0.0 
Perf Tools                          neuronperf-1.8.15.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.10.1.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.7.4.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.8.4.2.10.8.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.9.3.2.10.8.0 
TensorFlow                          tensorflow-neuron-2.10.1.2.10.8.0 
TensorFlow                          tensorflow-neuron-2.7.4.2.10.8.0 
TensorFlow                          tensorflow-neuron-2.8.4.2.10.8.0 
TensorFlow                          tensorflow-neuron-2.9.3.2.10.8.0 
PyTorch                             torch-neuron-1.10.2.2.9.17.0 
PyTorch                             torch-neuron-1.11.0.2.9.17.0 
PyTorch                             torch-neuron-1.12.1.2.9.17.0 
PyTorch                             torch-neuron-1.13.1.2.9.17.0 
PyTorch                             torch-neuron-1.9.1.2.9.17.0

Supported Python Versions for Inf1 packages#

List of packages in Neuron 2.17.0:

Package                                        Supported Python Versions              
dmlc_nnvm-1.18.2.0                                3.8, 3.9, 3.10 
dmlc_topi-1.18.2.0                                3.8, 3.9, 3.10 
dmlc_tvm-1.18.2.0                                 3.8, 3.9, 3.10 
inferentia_hwm-1.16.2.0                           3.8, 3.9, 3.10 
mx_neuron-1.8.0.2.4.40.0                          3.8, 3.9, 3.10 
mxnet_neuron-1.5.1.1.10.0.0                       3.8, 3.9, 3.10 
neuron-cc-1.21.0.0                                3.8, 3.9, 3.10 
neuronperf-1.8.15.0                               3.8, 3.9, 3.10 
tensorflow-neuron-2.10.1.2.10.8.0                 3.8, 3.9, 3.10 
tensorflow-neuron-2.7.4.2.10.8.0                  3.8, 3.9, 3.10 
tensorflow-neuron-2.8.4.2.10.8.0                  3.8, 3.9, 3.10 
tensorflow-neuron-2.9.3.2.10.8.0                  3.8, 3.9, 3.10 
torch-neuron-1.10.2.2.9.17.0                      3.8, 3.9, 3.10 
torch-neuron-1.11.0.2.9.17.0                      3.8, 3.9, 3.10 
torch-neuron-1.12.1.2.9.17.0                      3.8, 3.9, 3.10 
torch-neuron-1.13.1.2.9.17.0                      3.8, 3.9, 3.10 
torch-neuron-1.9.1.2.9.17.0                       3.8, 3.9, 3.10

Supported Python Versions for Inf2/Trn1 packages#

List of packages in Neuron 2.17.0:

Package                                        Supported Python Versions              
aws-neuronx-runtime-discovery-2.9                 3.8, 3.9, 3.10 
libneuronxla-2.0.755                              3.8, 3.9, 3.10 
libneuronxla-0.5.809                              3.8, 3.9, 3.10 
neuronx-cc-2.12.68.0                              3.8, 3.9, 3.10 
neuronx_distributed-0.6.0                         3.8, 3.9, 3.10 
neuronx_hwm-2.12.0.0                              3.8, 3.9, 3.10 
tensorflow-neuronx-2.10.1.2.1.0                   3.8, 3.9, 3.10 
tensorflow-neuronx-2.8.4.2.1.0                    3.8, 3.9, 3.10 
tensorflow-neuronx-2.9.3.2.1.0                    3.8, 3.9, 3.10 
torch-neuronx-1.13.1.1.13.1                       3.8, 3.9, 3.10 
torch-neuronx-2.0.0.2.0.1b0                       3.8, 3.9, 3.10 
torch-neuronx-2.1.1.2.0.1b0                       3.8, 3.9, 3.10 
torch_xla-1.13.1+torchneurond                     3.8, 3.9, 3.10 
torch_xla-2.1.1                                   3.8, 3.9, 3.10 
transformers-neuronx-0.9.474                      3.8, 3.9, 3.10

Supported Numpy Versions#

Neuron supports versions >= 1.21.6 and <= 1.22.2