This document is relevant for: Inf1, Inf2, Trn1, Trn1n

What’s New#

Table of contents

Neuron 2.10.0 (05/01/2023)
Previous Releases

Neuron 2.10.0 (05/01/2023)#

Table of contents

What’s New
Neuron Components Release Notes
Release Artifacts

What’s New #

This release introduces new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

What’s New	Details	Instances
Initial support for computer vision models inference	Added Stable Diffusion 2.1 model script for Text to Image Generation Added VGG model script for Image Classification Task Added UNet model script for Image Segmentation Task Please check aws-neuron-samples repository	Inf2, Trn1/Trn1n
Profiling support in PyTorch Neuron(`torch-neuronx`) for Inference with TensorBoard	See more at Profiling PyTorch Neuron (torch-neuronx) with TensorBoard	Inf2, Trn1/Trn1n
New Features and Performance Enhancements in transformers-neuronx	Support for the HuggingFace generate function. Model Serialization support for GPT2 models. (including model saving, loading, and weight swapping) Improved prompt context encoding performance. See Transformers Neuron (transformers-neuronx) Developer Guide for examples and usage See more at Transformers Neuron (transformers-neuronx) release notes	Inf2, Trn1/Trn1n
Support models larger than 2GB in TensorFlow 2.x Neuron (`tensorflow-neuronx`)	See Special Flags for details. (`tensorflow-neuronx`)	Trn1/Trn1n, Inf2
Support models larger than 2GB in TensorFlow 2.x Neuron (`tensorflow-neuron`)	See Special Flags for details. (`tensorflow-neuron`)	Inf1
Performance Enhancements in PyTorch C++ Custom Operators (Experimental)	Support for using multiple GPSIMD Cores in Custom C++ Operators See Custom Operators API Reference Guide [Experimental]	Trn1/Trn1n
Weight Deduplication Feature (Inf1)	Support for Sharing weights when loading multiple instance versions of the same model on different NeuronCores. See more at Neuron Runtime Configuration	Inf1
`nccom-test` - Collective Communication Benchmarking Tool	Supports enabling benchmarking sweeps on various Neuron Collective Communication operations. See NCCOM-TEST (Beta) for more details.	Trn1/Trn1n , Inf2
Announcing end of support for tensorflow-neuron 2.7 & mxnet-neuron 1.5 versions	See Announcing end of support for tensorflow-neuron versions 2.7 See Announcing end of support for mxnet-neuron versions 1.5	Inf1
Minor enhancements and bug fixes.	See Neuron Components Release Notes	Trn1/Trn1n , Inf2, Inf1
Release Artifacts	see Release Artifacts	Trn1/Trn1n , Inf2, Inf1

For more detailed release notes of the new features and resolved issues, see Neuron Components Release Notes.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see Model Architecture Fit Guidelines.

Neuron Components Release Notes #

Inf1, Trn1/Trn1n and Inf2 common packages #

Component	Instance/s	Package/s	Details
Neuron Runtime	Trn1/Trn1n, Inf1, Inf2	Trn1/Trn1n: `aws-neuronx-runtime-lib` (.deb, .rpm) Inf1: Runtime is linked into the ML frameworks packages	Neuron Runtime Release Notes
Neuron Runtime Driver	Trn1/Trn1n, Inf1, Inf2	`aws-neuronx-dkms` (.deb, .rpm)	Neuron Driver Release Notes
Neuron System Tools	Trn1/Trn1n, Inf1, Inf2	`aws-neuronx-tools` (.deb, .rpm)	Neuron System Tools
Containers	Trn1/Trn1n, Inf1, Inf2	`aws-neuronx-k8-plugin` (.deb, .rpm) `aws-neuronx-k8-scheduler` (.deb, .rpm) `aws-neuronx-oci-hooks` (.deb, .rpm)	Neuron K8 Release Notes Neuron Containers Release Notes
NeuronPerf (Inference only)	Trn1/Trn1n, Inf1, Inf2	`neuronperf` (.whl)	NeuronPerf 1.x Release Notes
TensorFlow Model Server Neuron	Trn1/Trn1n, Inf1, Inf2	`tensorflow-model-server-neuronx` (.deb, .rpm)	TensorFlow-Model-Server-Neuron (tensorflow-modeslserver-neuronx) Release Notes

Trn1/Trn1n and Inf2 only packages #

Component	Instance/s	Package/s	Details
PyTorch Neuron	Trn1/Trn1n, Inf2	`torch-neuronx` (.whl)	PyTorch Neuron (torch-neuronx) release notes PyTorch Neuron (torch-neuronx) - Supported Operators
TensorFlow Neuron	Trn1/Trn1n, Inf2	`tensorflow-neuronx` (.whl)	TensorFlow Neuron (tensorflow-neuronx) Release Notes
Neuron Compiler (Trn1/Trn1n, Inf2 only)	Trn1/Trn1n, Inf2	`neuronx-cc` (.whl)	Neuron Compiler (neuronx-cc) release notes
Collective Communication library	Trn1/Trn1n, Inf2	`aws-neuronx-collective` (.deb, .rpm)	Neuron Collectives Release Notes
Neuron Custom C++ Operators	Trn1/Trn1n, Inf2	`aws-neuronx-gpsimd-customop` (.deb, .rpm) `aws-neuronx-gpsimd-tools` (.deb, .rpm)	Neuron Custom C++ Library Release Notes Neuron Custom C++ Tools Release Notes
`transformers-neuronx`	Trn1/Trn1n, Inf2	GitHub repository (link)	Release Notes

Note

In next releases aws-neuronx-tools and aws-neuronx-runtime-lib will add support for Inf1.

Inf1 only packages #

Component	Instance/s	Package/s	Details
PyTorch Neuron	Inf1	`torch-neuron` (.whl)	PyTorch Neuron (torch-neuron) release notes PyTorch Neuron (torch-neuron) Supported operators
TensorFlow Neuron	Inf1	`tensorflow-neuron` (.whl)	TensorFlow Neuron (tensorflow-neuron (TF1.x)) Release Notes TensorFlow Neuron (tensorflow-neuron (TF1.x)) Supported operators TensorFlow Neuron (tensorflow-neuron (TF2.x)) Release Notes
Apache MXNet (Incubating)	Inf1	`mx_neuron` (.whl)	Apache MXNet Neuron (Incubating) Release Notes Neuron Apache MXNet (Incubating) Supported operators
Neuron Compiler (Inf1 only)	Inf1	`neuron-cc` (.whl)	Neuron Compiler (neuron-cc) for Inf1 Release Notes Neuron Supported operators

Release Artifacts #

Trn1 packages

List of packages in Neuron 2.10.0:

Component                           Package                                           
Collective Communication Library    aws-neuronx-collectives-2.13.7.0 
Driver                              aws-neuronx-dkms-2.9.4.0 
CustomOps                           aws-neuronx-gpsimd-customop-0.3.1.0 
CustomOps Tools                     aws-neuronx-gpsimd-tools-0.3.0.0 
General                             aws-neuronx-runtime-discovery-2.9 
Runtime Library                     aws-neuronx-runtime-lib-2.13.6.0 
System Tools                        aws-neuronx-tools-2.10.1.0 
General                             libneuronxla-0.5.207 
General                             neuronx_hwm-2.6.0.0 
PyTorch                             torch_xla-1.13.1 
Compiler                            neuronx-cc-2.6.0.19 
Kubernetes Plugin                   aws-neuronx-k8-plugin-2.13.4.0 
Kubernetes Scheduler                aws-neuronx-k8-scheduler-2.13.4.0 
OCI                                 aws-neuronx-oci-hook-2.2.0.0 
TensorFlow                          tensorflow-neuronx-2.10.1.2.1.0 
TensorFlow                          tensorflow-neuronx-2.7.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.8.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.9.3.2.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-1.15.0.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.10.1.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.7.4.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.8.4.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.9.3.2.8.1.0 
TensorBoard                         tensorboard-plugin-neuronx-2.5.26.0 
PyTorch                             torch-neuronx-1.13.1.1.7.0 
Transformers Neuron                 transformers-neuronx-0.3.32 
Compiler                            islpy-2021.1

Inf2 packages

List of packages in Neuron 2.10.0:

Component                           Package                                           
Collective Communication Library    aws-neuronx-collectives-2.13.7.0 
Driver                              aws-neuronx-dkms-2.9.4.0 
CustomOps                           aws-neuronx-gpsimd-customop-0.3.1.0 
CustomOps Tools                     aws-neuronx-gpsimd-tools-0.3.0.0 
General                             aws-neuronx-runtime-discovery-2.9 
Runtime Library                     aws-neuronx-runtime-lib-2.13.6.0 
System Tools                        aws-neuronx-tools-2.10.1.0 
General                             libneuronxla-0.5.207 
General                             neuronx_hwm-2.6.0.0 
PyTorch                             torch_xla-1.13.1 
Compiler                            neuronx-cc-2.6.0.19 
Kubernetes Plugin                   aws-neuronx-k8-plugin-2.13.4.0 
Kubernetes Scheduler                aws-neuronx-k8-scheduler-2.13.4.0 
OCI                                 aws-neuronx-oci-hook-2.2.0.0 
TensorFlow                          tensorflow-neuronx-2.10.1.2.1.0 
TensorFlow                          tensorflow-neuronx-2.7.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.8.4.2.1.0 
TensorFlow                          tensorflow-neuronx-2.9.3.2.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-1.15.0.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.10.1.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.7.4.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.8.4.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.9.3.2.8.1.0 
TensorBoard                         tensorboard-plugin-neuronx-2.5.26.0 
PyTorch                             torch-neuronx-1.13.1.1.7.0 
Transformers Neuron                 transformers-neuronx-0.3.32 
Compiler                            islpy-2021.1

Inf1 packages

List of packages in Neuron 2.10.0:

Component                           Package                                           
Driver                              aws-neuronx-dkms-2.9.4.0 
System Tools                        aws-neuronx-tools-2.10.1.0 
Compiler                            neuron-cc-1.15.0 
Kubernetes Plugin                   aws-neuronx-k8-plugin-2.13.4.0 
Kubernetes Scheduler                aws-neuronx-k8-scheduler-2.13.4.0 
OCI                                 aws-neuronx-oci-hook-2.2.0.0 
TensorFlow                          tensorflow-neuron-1.15.5.2.8.1.0 
TensorFlow                          tensorflow-neuron-2.10.1.2.8.1.0 
TensorFlow                          tensorflow-neuron-2.7.4.2.8.1.0 
TensorFlow                          tensorflow-neuron-2.8.4.2.8.1.0 
TensorFlow                          tensorflow-neuron-2.9.3.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-1.15.0.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.10.1.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.7.4.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.8.4.2.8.1.0 
TensorFlow Model Server             tensorflow-model-server-neuronx-2.9.3.2.8.1.0 
PyTorch                             torch-neuron-1.10.2.2.7.1.0 
PyTorch                             torch-neuron-1.11.0.2.7.1.0 
PyTorch                             torch-neuron-1.12.1.2.7.1.0 
PyTorch                             torch-neuron-1.13.1.2.7.1.0 
PyTorch                             torch-neuron-1.9.1.2.7.1.0 
MXNet                               mxnet_neuron-1.5.1.1.10.39.0 
MXNet                               mx_neuron-1.8.0.2.4.1.0 
Perf Tools                          neuronperf-1.8.0.0 
Runtime Library                     libnrt.so (Version 2.12.23.0)
Compiler                            dmlc_nnvm-1.15.0.0 
Compiler                            dmlc_topi-1.15.0.0 
Compiler                            dmlc_tvm-1.15.0.0 
Compiler                            inferentia_hwm-1.14.1 
Compiler                            islpy-2021.1

Previous Releases #

This document is relevant for: Inf1, Inf2, Trn1, Trn1n

AWS Neuron Documentation

What’s New

Contents

What’s New#

Neuron 2.10.0 (05/01/2023)#

What’s New #

Neuron Components Release Notes #

Inf1, Trn1/Trn1n and Inf2 common packages #

Trn1/Trn1n and Inf2 only packages #

Inf1 only packages #

Release Artifacts #

Previous Releases #