This document is relevant for: Trn2, Trn3

What is Neuron Observer?#

Neuron Observer collects accelerator and system telemetry from AWS Trainium clusters. It reports device health, hardware faults, and performance metrics for each node.

Note

Neuron Observer is in public beta for the Neuron 2.33 release. It is supported on Trainium 2 (Trn2) and Trainium 3 (Trn3) instances only. APIs, metric names, and installation steps can change before general availability. During the beta, neuron-monitor remains fully supported and no migration is required. For more information about neuron-monitor, see the neuron-monitor user guide.

Applies to#

Cluster health monitoring for training and inference workloads on Trainium.
Trainium 2 (Trn2) and Trainium 3 (Trn3) instances running on Amazon EKS.

Core capabilities#

Capability	Question it answers	Example output / detail
Health state	Is this device healthy?	3-state classification: `healthy`, `unhealthy`, or `unknown`.
Fault localization	Which node and device are affected?	Device health and fault events identify the affected node and device.
Performance telemetry	What accelerator and system signals are available for investigating performance?	Utilization, latency, EFA, ECC, runtime, and device metrics.
Lower-level API access	Can I access lower-level signals for custom scheduling, debugging, or research?	Snapshot API access to lower-level metrics and signals in addition to the default exported metric set.

How does Neuron Observer work?#

Neuron Observer runs a daemon on each Trainium node. The daemon updates device health when it receives driver health events and collects other telemetry at a configurable interval. Prometheus can scrape metrics and health gauges from the daemon, or the daemon can send them through OpenTelemetry. The API exposes fault events.

On a single node, the daemon collects telemetry from the Neuron driver, sysfs, procfs, and runtime data, stores the latest snapshot, and exposes it through the HTTP API, the nobs CLI, and an OpenTelemetry exporter that pushes to a co-located ADOT collector. The ADOT collector sends the metrics to your observability backends.

The daemon runs in a separate DaemonSet pod. If the daemon stops, the training workload continues, but Observer stops collecting and publishing node signals until the daemon recovers.

How you consume Neuron Observer signals#

Use Prometheus and Grafana, Amazon Managed Service for Prometheus (AMP), or Amazon CloudWatch to consume Neuron Observer signals. Use the API and CLI to integrate health signals into a scheduler, control plane, or debugging workflow.

What is Neuron Observer?

Contents