This document is relevant for: Inf1, Inf2, Trn1, Trn2, Trn3

NeuronX Runtime#

The NeuronX Runtime consists of a kernel driver and C/C++ libraries which provides APIs to access Inferentia and Trainium Neuron devices. The Neuron ML frameworks plugins for TensorFlow, PyTorch and Apache MXNet use the Neuron runtime to load and run models on the NeuronCores. Neuron runtime loads compiled deep learning models, also referred to as Neuron Executable File Format (NEFF) to the Neuron devices and is optimized for high-throughput and low-latency.

Get Started#

About the NeuronX Runtime

Learn about the AWS Neuron Runtime, its features, and capabilities for accessing Inferentia and Trainium Neuron devices.

Quickstart: Generate a Core Dump

Learn how to generate a Neuron runtime core dump for debugging runtime failures and analyzing device state.

Reference#

Runtime API Reference

Comprehensive guide to the Neuron Runtime API for developers building custom frameworks that call libnrt APIs directly.

Runtime Configuration

Learn how to configure the Neuron Runtime using environment variables to control NeuronCore allocation, logging, and more.

Troubleshooting on Inf1 and Trn1

Solutions for common issues encountered when using the Neuron Runtime on Inferentia and Trainium instances.

Frequently Asked Questions

Answers to common questions about the Neuron Runtime, including compatibility, configuration, and usage.

Learn More#

Runtime Configuration

Explore the Neuron Runtime

Deep dives into the Neuron Runtime, including NEFF files, compute-communication overlap, device memory, and core dumps.

Collectives#

About Collectives

Learn about Neuron Runtime collectives.

Deep Dive: Inter-node Collective Communication

Explore and understand techniques for communication across nodes in the Neuron Runtime.

Deep dive: Intra-node Collective Communication

Explore and understand techniques for communication within nodes in the Neuron Runtime.

Release Notes#

Runtime Release Notes

Latest updates, improvements, and bug fixes for the Neuron Runtime library.

Driver Release Notes

Release notes for the Neuron kernel driver that enables access to Neuron devices.

Collectives Release Notes

Release notes for the Neuron Collective Communication Library used for distributed training and inference.