This document is relevant for: Inf1, Trn1

Previous Releases Notes (Neuron 2.x)

Neuron 2.5.0 (11/23/2022)

Neuron 2.5.0 is a major release which introduces new features and resolves issues that improve stability for Inf1 customers.

Component	New in this release
PyTorch Neuron (torch-neuron)	PyTorch 1.12 support Python 3.8 support LSTM support on Inf1 R-CNN support on Inf1 Support for new API for core placement Support for improved logging Improved torch_neuron.trace() performance when using large graphs Reduced host memory usage of loaded models in libtorchneuron.so Additional operators support
TensorFlow Neuron (tensorflow-neuron)	tf-neuron-auto-multicore tool to enable automatic data parallel on multiple NeuronCores. Experimental support for tracing models larger than 2GB using extract-weights flag (TF2.x only), see TensorFlow 2.x (tensorflow-neuron) Tracing API tfn.auto_multicore Python API to enable automatic data parallel (TF2.x only)

This Neuron release is the last release that will include torch-neuron versions 1.7 and 1.8, and that will include tensorflow-neuron versions 2.5 and 2.6.

In addition, this release introduces changes to the Neuron packaging and installation instructions for Inf1 customers, see Introducing Neuron packaging and installation changes for Inf1 customers for more information.

For more detailed release notes of the new features and resolved issues, see Neuron Components Release Notes.

Neuron 2.4.0 (10/27/2022)

This release introduces new features and resolves issues that improve stability. The release introduces “memory utilization breakdown” feature in both Neuron Monitor and Neuron Top system tools. The release introduces support for “NeuronCore Based Sheduling” capability to the Neuron Kubernetes Scheduler and introduces new operators support in Neuron Compiler and PyTorch Neuron. This release introduces also additional eight (8) samples of models’ fine tuning using PyTorch Neuron. The new samples can be found in the AWS Neuron Samples GitHub repository.

Neuron 2.3.0 (10/10/2022)

Overview

This Neuron 2.3.0 release extends Neuron 1.x and adds support for the new AWS Trainium powered Amazon EC2 Trn1 instances. With this release, you can now run deep learning training workloads on Trn1 instances to save training costs by up to 50% over equivalent GPU-based EC2 instances, while getting the highest training performance in AWS cloud for popular NLP models.

What’s New	New features and capabilities Introducing Packaging and installation changes Introducing New Neuron GitHub Repositories Announcing Neuron Runtime 1.x (neuron-rtd) end-of-support
Tested workloads and known issues	Tested Workloads Known Issues

New features and capabilities

Neural-networks training support

• Supported instances: Trn1

• Supported Frameworks: PyTorch Neuron (torch-neuronx)

• Supported Data-types

  • FP32, BF16

• Supported Rounding Modes

  • Stochastic Rounding (SR)

  • Round Nearest ties to Even (RNE)

• Supported Automatic Casting Methods

  • Neuron automatic casting of FP32 tensors / weights / operations to BF16 - Default mode

  • PyTorch automatic casting

  • Full BF16 automatic casting (via XLA_USE_BF16=1 environment variable)

PyTorch Neuron (torch-neuronx)

• PyTorch 1.11

• Supported instances: Trn1

• Supported Python versions: Python 3.7, Python 3.8

• Eager Debug Mode

• Persistent Cache for compilation

• Collective compute operations: AllReduce

• Optimizers: AdamW, SGD

• Tested loss functions: Negative log-likelihood (NLL), Cross-entropy

• Training Libraries/Frameworks

  • torch.distributed

  • Megatron-LM Reference for Neuron

  • Training Examples

    • Multi-Layer Perceptron Training Tutorial

    • Hugging Face BERT Pretraining Tutorial

    • PyTorch Neuron for Trainium Hugging Face BERT MRPC task finetuning using Hugging Face Trainer API

    • Megatron-LM GPT Pretraining Tutorial

• For More information:

  • PyTorch Neuron

  • Developer Guide for Training with PyTorch Neuron (torch-neuronx)

  • PyTorch Neuron (torch-neuronx) release notes

  • PyTorch Neuron (torch-neuronx) for Training Troubleshooting Guide

Neuron Runtime, Drivers and Networking Components

• Neuron Runtime 2.9

• Supported instances: Trn1, Inf1

• Elastic Fabric Adapter (EFA) @ 800Gbps

• Collective communication operators

  • AllReduce

  • AllGather

  • ReduceScatter

• Release Notes:

  • Neuron Runtime Release Notes

  • Neuron Driver Release Notes

Neuron Tools

• Neuron system tools - Adding Trn1 support to the following tools:

  • neuron-monitor

  • neuron-top

  • neuron-ls

• Release Notes:

  • Neuron System Tools

Developer Flows

• Containers

  • Deep Learning Containers (DLC) supporting PyTorch Neuron (torch-neuronx)

• Multi-Instance distributed workloads orchestration:

  • AWS ParallelCluster (Through custom AMI build)

  • Amazon Elastic Compute Cloud (ECS)

• Supported Amazon Linux Images (AMIs)

  • Ubuntu 20 Neuron DLAMI-base (Python 3.8)

  • Amazon Linux2 Neuron DLAMI-base (Python 3.7)

  • Ubuntu 18 Neuron DLAMI-base (Python 3.7)

  • Ubuntu 18 AMI (Python 3.7)

  • Ubuntu 20 AMI (Python 3.8)

  • Amazon Linux2 AMI (Python 3.7)

Tested Workloads

The following workloads were tested in this release:

• Distributed data-parallel pre-training of Hugging Face BERT model on single Trn1.32xl instance (32 NeuronCores).

• Distributed data-parallel pre-training of Hugging Face BERT model on multiple Trn1.32xl instances.

• HuggingFace BERT MRPC task finetuning on single NeuronCore or multiple NeuronCores (data-parallel).

• Megatron-LM GPT3 (6.7B parameters) pre-training on single Trn1.32xl instance.

• Megatron-LM GPT3 (6.7B parameters) pre-training on multi Trn1.32xl instances.

• Multi-Layer Perceptron (ML) model training on single NeuronCore or multiple NeuronCores (data-parallel).

Known Issues

• For maximum training performance, please set environment variables XLA_USE_BF16=1 to enable full BF16 and Stochastic Rounding.

This document is relevant for: Inf1, Trn1