This document is relevant for: Inf1, Inf2, Trn1, Trn1n

What’s New

Neuron 2.9.1 (04/19/2023)

Minor patch release to add support for deserialized torchscript model compilation and support for multi-node training in EKS. Fixes included in this release are critical to enable training and deploying models with Amazon Sagemaker or Amazon EKS.

Neuron 2.9.0 (03/28/2023)

What’s New

This release adds support for EC2 Trn1n instances, introduces new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

What’s New	Details	Instances
Support for EC2 Trn1n instances	Updated Neuron Runtime for Trn1n instances Overall documentation update to include Trn1n instances	Trn1n
New Analyze API in PyTorch Neuron (torch-neuronx)	A new API that return list of supported and unsupported PyTorch operators for a model. See PyTorch Neuron (torch-neuronx) Analyze API for Inference	Trn1, Inf2
Support models that are larger than 2GB in PyTorch Neuron (torch-neuron) on Inf1	See separate_weights flag to torch_neuron.trace() to support models that are larger than 2GB	Inf1
Performance Improvements	Up to 10% higher throughput when training GPT3 6.7B model on multi-node	Trn1
Dynamic Batching support in TensorFlow 2.x Neuron (tensorflow-neuronx)	See Special Flags for details.	Trn1, Inf2
NeuronPerf support for Trn1/Inf2 instances	Added Trn1/Inf2 support for PyTorch Neuron (torch-neuronx) and TensorFlow 2.x Neuron (tensorflow-neuronx)	Trn1, Inf2
Hierarchical All-Reduce and Reduce-Scatter collective communication	Added support for hierarchical All-Reduce and Reduce-Scatter in Neuron Runtime to enable better scalability of distributed workloads .	Trn1, Inf2
New Tutorials added	Added tutorial to fine-tune T5 model Added tutorial to demonstrate use of Libtorch with PyTorch Neuron (torch-neuronx) for inference [html]	Trn1, Inf2
Minor enhancements and bug fixes.	See Neuron Components Release Notes	Trn1, Inf2, Inf1
Release included packages	see Release Content	Trn1, Inf2, Inf1

For more detailed release notes of the new features and resolved issues, see Neuron Components Release Notes.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see Model Architecture Fit Guidelines.

Neuron Components Release Notes

Inf1, Trn1/Trn1n and Inf2 common packages

Component	Instance/s	Package/s	Details
Neuron Runtime	Trn1/Trn1n, Inf1, Inf2	Trn1/Trn1n: aws-neuronx-runtime-lib (.deb, .rpm) Inf1: Runtime is linked into the ML frameworks packages	Neuron Runtime Release Notes
Neuron Runtime Driver	Trn1/Trn1n, Inf1, Inf2	aws-neuronx-dkms (.deb, .rpm)	Neuron Driver Release Notes
Neuron System Tools	Trn1/Trn1n, Inf1, Inf2	aws-neuronx-tools (.deb, .rpm)	Neuron System Tools
Containers	Trn1/Trn1n, Inf1, Inf2	aws-neuronx-k8-plugin (.deb, .rpm) aws-neuronx-k8-scheduler (.deb, .rpm) aws-neuronx-oci-hooks (.deb, .rpm)	Neuron K8 Release Notes Neuron Containers Release Notes
NeuronPerf (Inference only)	Trn1/Trn1n, Inf1, Inf2	neuronperf (.whl)	NeuronPerf 1.x Release Notes
TensorFlow Model Server Neuron	Trn1/Trn1n, Inf1, Inf2	tensorflow-model-server-neuronx (.deb, .rpm)	TensorFlow-Model-Server-Neuron (tensorflow-modeslserver-neuronx) Release Notes

Trn1/Trn1n and Inf2 only packages

Component	Instance/s	Package/s	Details
PyTorch Neuron	Trn1/Trn1n, Inf2	torch-neuronx (.whl)	PyTorch Neuron (torch-neuronx) release notes PyTorch Neuron (torch-neuronx) - Supported Operators
TensorFlow Neuron	Trn1/Trn1n, Inf2	tensorflow-neuronx (.whl)	TensorFlow Neuron (tensorflow-neuronx) Release Notes
Neuron Compiler (Trn1/Trn1n, Inf2 only)	Trn1/Trn1n, Inf2	neuronx-cc (.whl)	Neuron Compiler (neuronx-cc) release notes
Collective Communication library	Trn1/Trn1n, Inf2	aws-neuronx-collective (.deb, .rpm)	Neuron Collectives Release Notes
Neuron Custom C++ Operators	Trn1/Trn1n, Inf2	aws-neuronx-gpsimd-customop (.deb, .rpm) aws-neuronx-gpsimd-tools (.deb, .rpm)	Neuron Custom C++ Library Release Notes Neuron Custom C++ Tools Release Notes
transformers-neuronx	Trn1/Trn1n, Inf2	GitHub repository (link)	Release Notes

Note

In next releases aws-neuronx-tools and aws-neuronx-runtime-lib will add support for Inf1.

Inf1 only packages

Component	Instance/s	Package/s	Details
PyTorch Neuron	Inf1	torch-neuron (.whl)	PyTorch Neuron (torch-neuron) release notes PyTorch Neuron (torch-neuron) Supported operators
TensorFlow Neuron	Inf1	tensorflow-neuron (.whl)	TensorFlow Neuron (tensorflow-neuron (TF1.x)) Release Notes TensorFlow Neuron (tensorflow-neuron (TF1.x)) Supported operators TensorFlow Neuron (tensorflow-neuron (TF2.x)) Release Notes
Apache MXNet (Incubating)	Inf1	mx_neuron (.whl)	Apache MXNet Neuron (Incubating) Release Notes Neuron Apache MXNet (Incubating) Supported operators
Neuron Compiler (Inf1 only)	Inf1	neuron-cc (.whl)	Neuron Compiler (neuron-cc) for Inf1 Release Notes Neuron Supported operators

Release Content

• Release Content

Previous Releases

• Previous Releases Notes (Neuron 2.x)

• Previous Release Notes (Neuron 1.x)

This document is relevant for: Inf1, Inf2, Trn1, Trn1n