What’s New
Contents
This document is relevant for: Inf1
, Inf2
, Trn1
, Trn1n
What’s New#
Table of contents
Neuron 2.17.0 (02/13/2024)#
What’s New#
Neuron 2.17 release improves small collective communication operators (smaller than 16MB) by up to 30%, which improves large language model (LLM) Inference performance by up to 10%. This release also includes improvements in Neuron Profiler and other minor enhancements and bug fixes.
For more detailed release notes of the new features and resolved issues, see Neuron Components Release Notes.
To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see Model Architecture Fit Guidelines.
Neuron Components Release Notes#
Inf1, Trn1/Trn1n and Inf2 common packages#
Component |
Instance/s |
Package/s |
Details |
---|---|---|---|
Neuron Runtime |
Trn1/Trn1n, Inf1, Inf2 |
|
|
Neuron Runtime Driver |
Trn1/Trn1n, Inf1, Inf2 |
|
|
Neuron System Tools |
Trn1/Trn1n, Inf1, Inf2 |
|
|
Containers |
Trn1/Trn1n, Inf1, Inf2 |
|
|
NeuronPerf (Inference only) |
Trn1/Trn1n, Inf1, Inf2 |
|
|
TensorFlow Model Server Neuron |
Trn1/Trn1n, Inf1, Inf2 |
|
|
Neuron Documentation |
Trn1/Trn1n, Inf1, Inf2 |
Trn1/Trn1n and Inf2 only packages#
Component |
Instance/s |
Package/s |
Details |
---|---|---|---|
PyTorch Neuron |
Trn1/Trn1n, Inf2 |
|
|
TensorFlow Neuron |
Trn1/Trn1n, Inf2 |
|
|
Neuron Compiler (Trn1/Trn1n, Inf2 only) |
Trn1/Trn1n, Inf2 |
|
|
Collective Communication library |
Trn1/Trn1n, Inf2 |
|
|
Neuron Custom C++ Operators |
Trn1/Trn1n, Inf2 |
|
|
Transformers Neuron |
Trn1/Trn1n, Inf2 |
|
|
Neuron Distributed |
Trn1/Trn1n, Inf2 |
|
|
AWS Neuron Reference for NeMo Megatron |
Trn1/Trn1n |
Note
In next releases aws-neuronx-tools
and aws-neuronx-runtime-lib
will add support for Inf1.
Inf1 only packages#
Component |
Instance/s |
Package/s |
Details |
---|---|---|---|
PyTorch Neuron |
Inf1 |
|
|
TensorFlow Neuron |
Inf1 |
|
|
Apache MXNet |
Inf1 |
|
|
Neuron Compiler (Inf1 only) |
Inf1 |
|
Release Artifacts#
Table of contents
Trn1 packages#
List of packages in Neuron 2.17.0:
Component Package
Collective Communication Library aws-neuronx-collectives-2.20.11.0
Driver aws-neuronx-dkms-2.15.9.0
nan aws-neuronx-gpsimd-customop-lib-0.9.1.0
CustomOps Tools aws-neuronx-gpsimd-tools-0.9.0.0
Kubernetes Plugin aws-neuronx-k8-plugin-2.19.16.0
Kubernetes Scheduler aws-neuronx-k8-scheduler-2.19.16.0
OCI aws-neuronx-oci-hook-2.2.45.0
General aws-neuronx-runtime-discovery-2.9
Runtime Library aws-neuronx-runtime-lib-2.20.11.0
System Tools aws-neuronx-tools-2.17.0.0
Framework libneuronxla-2.0.755
Framework libneuronxla-0.5.809
Compiler neuronx-cc-2.12.68.0
Neuron Distributed neuronx_distributed-0.6.0
Compiler neuronx_hwm-2.12.0.0
TensorBoard tensorboard-plugin-neuronx-2.6.1.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.10.1.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.7.4.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.8.4.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.9.3.2.10.8.0
TensorFlow tensorflow-neuronx-2.10.1.2.1.0
TensorFlow tensorflow-neuronx-2.7.4.2.1.0
TensorFlow tensorflow-neuronx-2.8.4.2.1.0
TensorFlow tensorflow-neuronx-2.9.3.2.1.0
PyTorch torch-neuronx-1.13.1.1.13.1
PyTorch torch-neuronx-2.0.0.2.0.1b0
PyTorch torch-neuronx-2.1.1.2.0.1b0
PyTorch torch_xla-1.13.1+torchneurond
PyTorch torch_xla-2.1.1
Transformers Neuron transformers-neuronx-0.9.474
Inf2 packages#
List of packages in Neuron 2.17.0:
Component Package
Collective Communication Library aws-neuronx-collectives-2.20.11.0
Driver aws-neuronx-dkms-2.15.9.0
nan aws-neuronx-gpsimd-customop-lib-0.9.1.0
CustomOps Tools aws-neuronx-gpsimd-tools-0.9.0.0
Kubernetes Plugin aws-neuronx-k8-plugin-2.19.16.0
Kubernetes Scheduler aws-neuronx-k8-scheduler-2.19.16.0
OCI aws-neuronx-oci-hook-2.2.45.0
General aws-neuronx-runtime-discovery-2.9
Runtime Library aws-neuronx-runtime-lib-2.20.11.0
System Tools aws-neuronx-tools-2.17.0.0
Framework libneuronxla-2.0.755
Framework libneuronxla-0.5.809
Compiler neuronx-cc-2.12.68.0
Neuron Distributed neuronx_distributed-0.6.0
Compiler neuronx_hwm-2.12.0.0
TensorBoard tensorboard-plugin-neuronx-2.6.1.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.10.1.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.7.4.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.8.4.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.9.3.2.10.8.0
TensorFlow tensorflow-neuronx-2.10.1.2.1.0
TensorFlow tensorflow-neuronx-2.8.4.2.1.0
TensorFlow tensorflow-neuronx-2.9.3.2.1.0
PyTorch torch-neuronx-1.13.1.1.13.1
PyTorch torch-neuronx-2.0.0.2.0.1b0
PyTorch torch-neuronx-2.1.1.2.0.1b0
PyTorch torch_xla-1.13.1+torchneurond
PyTorch torch_xla-2.1.1
Transformers Neuron transformers-neuronx-0.9.474
Inf1 packages#
List of packages in Neuron 2.17.0:
Component Package
Driver aws-neuronx-dkms-2.15.9.0
Kubernetes Plugin aws-neuronx-k8-plugin-2.19.16.0
Kubernetes Scheduler aws-neuronx-k8-scheduler-2.19.16.0
OCI aws-neuronx-oci-hook-2.2.45.0
System Tools aws-neuronx-tools-2.17.0.0
Compiler dmlc_nnvm-1.18.2.0
Compiler dmlc_topi-1.18.2.0
Compiler dmlc_tvm-1.18.2.0
Compiler inferentia_hwm-1.16.2.0
MXNet mx_neuron-1.8.0.2.4.40.0
MXNet mxnet_neuron-1.5.1.1.10.0.0
Compiler neuron-cc-1.21.0.0
Perf Tools neuronperf-1.8.15.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.10.1.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.7.4.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.8.4.2.10.8.0
TensorFlow Model Server tensorflow-model-server-neuronx-2.9.3.2.10.8.0
TensorFlow tensorflow-neuron-2.10.1.2.10.8.0
TensorFlow tensorflow-neuron-2.7.4.2.10.8.0
TensorFlow tensorflow-neuron-2.8.4.2.10.8.0
TensorFlow tensorflow-neuron-2.9.3.2.10.8.0
PyTorch torch-neuron-1.10.2.2.9.17.0
PyTorch torch-neuron-1.11.0.2.9.17.0
PyTorch torch-neuron-1.12.1.2.9.17.0
PyTorch torch-neuron-1.13.1.2.9.17.0
PyTorch torch-neuron-1.9.1.2.9.17.0
Supported Python Versions for Inf1 packages#
List of packages in Neuron 2.17.0:
Package Supported Python Versions
dmlc_nnvm-1.18.2.0 3.8, 3.9, 3.10
dmlc_topi-1.18.2.0 3.8, 3.9, 3.10
dmlc_tvm-1.18.2.0 3.8, 3.9, 3.10
inferentia_hwm-1.16.2.0 3.8, 3.9, 3.10
mx_neuron-1.8.0.2.4.40.0 3.8, 3.9, 3.10
mxnet_neuron-1.5.1.1.10.0.0 3.8, 3.9, 3.10
neuron-cc-1.21.0.0 3.8, 3.9, 3.10
neuronperf-1.8.15.0 3.8, 3.9, 3.10
tensorflow-neuron-2.10.1.2.10.8.0 3.8, 3.9, 3.10
tensorflow-neuron-2.7.4.2.10.8.0 3.8, 3.9, 3.10
tensorflow-neuron-2.8.4.2.10.8.0 3.8, 3.9, 3.10
tensorflow-neuron-2.9.3.2.10.8.0 3.8, 3.9, 3.10
torch-neuron-1.10.2.2.9.17.0 3.8, 3.9, 3.10
torch-neuron-1.11.0.2.9.17.0 3.8, 3.9, 3.10
torch-neuron-1.12.1.2.9.17.0 3.8, 3.9, 3.10
torch-neuron-1.13.1.2.9.17.0 3.8, 3.9, 3.10
torch-neuron-1.9.1.2.9.17.0 3.8, 3.9, 3.10
Supported Python Versions for Inf2/Trn1 packages#
List of packages in Neuron 2.17.0:
Package Supported Python Versions
aws-neuronx-runtime-discovery-2.9 3.8, 3.9, 3.10
libneuronxla-2.0.755 3.8, 3.9, 3.10
libneuronxla-0.5.809 3.8, 3.9, 3.10
neuronx-cc-2.12.68.0 3.8, 3.9, 3.10
neuronx_distributed-0.6.0 3.8, 3.9, 3.10
neuronx_hwm-2.12.0.0 3.8, 3.9, 3.10
tensorflow-neuronx-2.10.1.2.1.0 3.8, 3.9, 3.10
tensorflow-neuronx-2.8.4.2.1.0 3.8, 3.9, 3.10
tensorflow-neuronx-2.9.3.2.1.0 3.8, 3.9, 3.10
torch-neuronx-1.13.1.1.13.1 3.8, 3.9, 3.10
torch-neuronx-2.0.0.2.0.1b0 3.8, 3.9, 3.10
torch-neuronx-2.1.1.2.0.1b0 3.8, 3.9, 3.10
torch_xla-1.13.1+torchneurond 3.8, 3.9, 3.10
torch_xla-2.1.1 3.8, 3.9, 3.10
transformers-neuronx-0.9.474 3.8, 3.9, 3.10
Supported Numpy Versions#
Neuron supports versions >= 1.21.6 and <= 1.22.2