This document is relevant for: Inf1, Inf2, Trn1, Trn2, Trn3

Troubleshooting guide

Use this page to find the troubleshooting guide for the Neuron component that you use. Each guide lists symptoms, causes, and fixes for that component.

Setup and installation

• Installation troubleshooting - Package installation, import errors, and driver issues

Training

• Troubleshooting: TorchNeuron - Installation, eager mode, torch.compile, and distributed training issues with native PyTorch on Trainium

• Troubleshooting: TorchTitan Neuron - Common TorchTitan errors and the TorchTitan profiler interface

Inference

• Debugging model code in vLLM Neuron - Debugging techniques for models that run on vLLM Neuron

• Debugging accuracy issues in vLLM Neuron - Find where accuracy drift starts in a vLLM Neuron deployment

• Accuracy evaluation and debugging in vLLM Omni Neuron - Evaluate output quality and isolate accuracy issues in diffusion models

Kernels (NKI)

• NKI FAQ - Common questions and issues when you write and run NKI kernels

Compiler

• Neuron compiler error codes - Causes and fixes for neuronx-cc compiler errors

Runtime

• Neuron Runtime troubleshooting - Driver installation, model load, memory, hardware, and collective communication errors

• nrtpy troubleshooting - Common issues with the nrtpy Python runtime layer

• Neuron Runtime core dumps - Capture and inspect core dumps when a model fails on the device

Developer tools

• Neuron Explorer troubleshooting and FAQ - Profile capture, processing, and missing data issues

• Neuron Observer troubleshooting and FAQs - Installation and metrics export issues on Amazon EKS

Containers and deployment

• Troubleshooting Neuron containers - Runtime, driver, and device access errors in Neuron Deep Learning Containers

• Neuron containers FAQ - Common questions about Neuron containers

JAX

• JAX NeuronX known issues - Known issues and limitations for JAX on Neuron

Legacy components

• NxD Inference troubleshooting - Troubleshooting for the legacy NxD Inference library

Troubleshooting guides for retired components, such as torch-neuron, TensorFlow Neuron, MXNet Neuron, NeuronPerf, and NxD Training, are in the archive.

Get help

If your issue is not covered in these guides, submit a request with AWS Neuron GitHub issues or contact AWS Support.

This document is relevant for: Inf1, Inf2, Trn1, Trn2, Trn3