How to isolate faulty Neuron devices on Amazon EKS with device taints

Note

Neuron device taints are a beta feature. Details on this page may change before general availability.

Task overview

This topic shows how to use Neuron device taints on Amazon EKS to isolate an individual faulty Neuron device, or a faulty node, so that the Kubernetes scheduler keeps new work off it and can evict running work onto healthy devices. Device taints build on the Neuron DRA driver and work only when the DRA driver manages your Neuron devices.

What are device taints?

A device taint is a Kubernetes mechanism that marks an individual accelerator device as unavailable for scheduling, much like a node taint marks a whole node. Each taint has a key, a value, and an effect that tells the scheduler to keep new work off the device and, optionally, to evict work already running on it. Dynamic Resource Allocation (DRA) drivers can apply device taints to the devices they manage, and cluster admins can apply them across a cluster with a DeviceTaintRule. Neuron device taints apply this mechanism to Neuron devices on Trainium platforms.

For more information, see Device taints and tolerations in the Kubernetes documentation.

How device taint isolation works

Neuron Observer runs on each Neuron node as a DaemonSet and watches the Neuron device driver’s per-device health signals. On a health change, Observer pushes an event to the Neuron DRA driver, which learns of the fault within about a second and adds a taint to that device (or, on UltraServers, to the node). A kernel hardware-error event surfaced through Observer can taint a device within a subsecond. The Kubernetes scheduler then acts on the taint. The isolation granularity depends on the instance type:

• Standalone Trainium instances: the DRA driver taints only the faulty device. The node’s other devices stay schedulable.

• UltraServer instances (trn2u.*, trn3.*): a claim takes a node’s devices together, so the DRA driver taints the whole node. The other nodes in the UltraServer stay schedulable. No configuration is needed — the DRA driver selects device-level or node-level tainting automatically from the instance type.

All device-health taints use the single key neuron.aws.com/device-unhealthy. The taint value names the specific fault, so an operator can tell from the ResourceSlice what happened. Taint effects are of three types:

• NoSchedule: Blocks new workloads; leaves running ones alone.

• NoExecute: Blocks new workloads and evicts running ones.

• None: Purely an informational taint on the ResourceSlice. It does not disrupt current/new workloads.

Workloads can opt out of a taint with a toleration. A toleration on the claim request lets a pod keep using a tainted device, and tolerationSeconds gives it a grace period before eviction. See Tolerate a fault in your workload for more details.

Prerequisites

• Kubernetes version: Kubernetes 1.36 or later on the control plane, with the node AMI on a matching version. The admin-set DeviceTaintRule API requires Kubernetes 1.37 or later; see Set manual device taints.

• Neuron DRA driver: Install and configure the Neuron DRA driver (version 1.3.0 or later). Device taints work only when the DRA driver manages your Neuron devices.

• Neuron driver: Use the Neuron driver from Neuron SDK 2.33.0 or later, which provides the per-device health signals this feature consumes. On an earlier driver, the DRA driver logs a warning and disables health monitoring.

• Neuron Observer: Deploy the Neuron Observer DaemonSet to your cluster. Observer ships with Neuron SDK 2.33.0 — the same release as the required Neuron driver — and watches the driver’s per-device health signals, pushing fault events to the DRA driver.

Enable device health monitoring

1: Enable health monitoring on the DRA driver.

Health monitoring is off by default. Enable it with the Neuron Helm chart:

helm upgrade --install neuron-helm-chart oci://public.ecr.aws/neuron/neuron-helm-chart \
  --set "draDriver.enabled=true" \
  --set "draDriver.deviceHealthMonitoring.enabled=true"

2: Confirm the health monitor started.

Check the DRA driver logs and confirm it is consuming Neuron Observer events:

kubectl logs -n neuron-dra-driver <driver-pod> | grep Observer
# NeuronObserver event stream consumer started

What you’ll see when a device faults

When Neuron Observer reports a device fault, the DRA driver taints the affected device or node automatically, with no action required from you. What you see depends on the instance type.

On a standalone Trainium instance

If an uncorrectable HBM error occurs on device 3, the DRA driver taints just that device on the first Observer event that reports it.

What you’ll see: A pod starts terminating on its own.

kubectl get pods -w
NAME    READY   STATUS        RESTARTS   AGE
pod0    1/1     Terminating   0          4m12s

Why: An uncorrectable HBM error hit device 3. The DRA driver tainted that one device the moment Neuron Observer reported it.

kubectl logs -n neuron-dra-driver -l app=neuron-dra-driver-kubelet-plugin | grep "Applied device taint"

"Applied device taint from NeuronObserver FaultEvent" deviceID=3 \
  taintKey="neuron.aws.com/device-unhealthy" taintValue="hbm-uncorrectable-error" \
  taintEffect="NoExecute" components=["hbm"] suggestedAction="DO_NOT_USE" latched=true

Confirm it yourself: The taint is on the device in the node’s ResourceSlice.

kubectl get resourceslices -o json \
  | jq '[.items[].spec.devices[]? | select(.taints != null) | {name, taints}]'

[
  {
    "name": "neuron-device-3",
    "taints": [
      {
        "key": "neuron.aws.com/device-unhealthy",
        "value": "hbm-uncorrectable-error",
        "effect": "NoExecute",
        "timeAdded": "2026-10-15T14:32:01Z"
      }
    ]
  }
]

For a NoExecute taint, recycle the node to return all of its devices to a healthy, schedulable state.

On an UltraServer

UltraServer instances group Trainium nodes together. Each node still runs its own DRA driver pod and monitors its own devices. However, because a claim takes the whole node’s devices together, the node advertises a single neuron-node resource for all of them. That one resource is the only place a taint can land, so tainting is at node granularity: fault on any device affects the whole node.

What you’ll see: The job holding the node starts terminating.

kubectl get pods -w
NAME    READY   STATUS        RESTARTS   AGE
pod0    1/1     Terminating   0          4m12s

Why: An uncorrectable HBM error hit device 3. The driver log still names the device that faulted, so you can tell which one caused it.

kubectl logs -n neuron-dra-driver -l app=neuron-dra-driver-kubelet-plugin | grep "Applied device taint"

"Applied device taint from NeuronObserver FaultEvent" deviceID=3 \
  taintKey="neuron.aws.com/device-unhealthy" taintValue="hbm-uncorrectable-error" \
  taintEffect="NoExecute" components=["hbm"] suggestedAction="DO_NOT_USE" latched=true

Confirm it yourself: The taint is on neuron-node rather than on an individual device.

kubectl get resourceslices -o json \
  | jq '[.items[].spec.devices[]? | select(.taints != null) | {name, taints}]'

[
  {
    "name": "neuron-node",
    "taints": [
      {
        "key": "neuron.aws.com/device-unhealthy",
        "value": "hbm-uncorrectable-error",
        "effect": "NoExecute",
        "timeAdded": "2026-10-15T14:32:01Z"
      }
    ]
  }
]

What this means in practice:

• Recovery is per fault, not per device. The taint is removed only when no device on the node still reports or latches that same key and value. One device clearing while a sibling is still faulted leaves the taint in place.

• Eviction moves work to healthy nodes, not healthy devices.

• Peer nodes are flagged, not taken out of service. When a node in the same UltraServer goes unhealthy, each surviving node applies neuron.aws.com/peer-tainted to its own neuron-node resource. The effect is None: it does not block scheduling and does not evict anything. Peer state is polled, so expect up to about 15 seconds before it appears or clears. Peer tainting requires the Neuron UltraServer Operator.

Tolerate a fault in your workload

By default, a NoExecute taint evicts your pod as soon as it lands. A toleration with tolerationSeconds delays that eviction, giving your workload a window to save state and exit on its own terms. Add a toleration to the device request in your ResourceClaimTemplate:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: fault-tolerant-neurons
spec:
  spec:
    devices:
      requests:
      - name: neurons
        exactly:
          deviceClassName: neuron.aws.com
          allocationMode: All
          tolerations:
          - key: neuron.aws.com/device-unhealthy
            operator: Exists
            effect: NoExecute
            tolerationSeconds: 120   # 2 minutes to checkpoint before eviction

With tolerationSeconds set, the pod gets that many seconds after a taint lands before eviction. Omit tolerationSeconds to tolerate the fault indefinitely. A NoSchedule taint never evicts a running pod, so most workloads need no toleration for it.

Note

The grace period lets a pod shut down gracefully. Whether it can also checkpoint depends on the fault: after a data-integrity fault such as an uncorrectable memory error, the device state may be unreliable, so a checkpoint taken at that point may not be trustworthy.

Confirm your work

To confirm health monitoring is active, check that the DRA driver is consuming Neuron Observer events:

kubectl logs -n neuron-dra-driver <driver-pod> | grep Observer

Fault-to-effect mapping

Every device-health taint uses the single key neuron.aws.com/device-unhealthy.

The following table lists the default fault-to-effect mapping the Neuron DRA driver applies. To change it, see Override the fault-to-effect mapping.

Fault	Effect	Taint value
hbm	NoExecute	hbm-uncorrectable-error
sram	NoExecute	sram-uncorrectable-error
device_unreachable	NoExecute	device-unreachable
firmware_service_api	NoSchedule	firmware-service-api-error
firmware_service_processor	NoSchedule	firmware-service-processor-error
link_host	NoExecute	link-host-error
link_intraserver	NoSchedule	link-intraserver-error
link_interserver	NoSchedule	link-interserver-error
link_efa	NoSchedule	link-efa-error
link_switch	NoSchedule	link-switch-error
fabric_error	NoExecute	fabric-error

Faults that compromise running work map to NoExecute (eviction): uncorrectable memory errors, a host-link error, a fabric error, and an unreachable device. The remaining interconnect and firmware faults map to NoSchedule (cordon): the device may still compute correctly, so the driver keeps new work off it without the disruption of eviction.

Override the fault-to-effect mapping

Override the default mapping to control how disruptive each fault is in your cluster.

Create a taint-policy ConfigMap.

List only the faults you want to change; any fault you omit keeps its default effect.

apiVersion: v1
kind: ConfigMap
metadata:
  name: neuron-dra-taint-policy
  namespace: neuron-dra-driver
data:
  policy.yaml: |
    policies:
      # De-escalate: cordon on an HBM fault instead of evicting.
      hbm:
        effect: NoSchedule
      # Escalate: evict on an EFA link fault instead of cordoning.
      # Escalation requires explicit opt-in.
      link_efa:
        effect: NoExecute
        allowEscalation: true

Apply it:

kubectl apply -f neuron-dra-taint-policy.yaml

Making a fault less disruptive than its default is allowed directly. Making a fault more disruptive requires allowEscalation: true on that entry, so a configuration change cannot cause fleet-wide evictions by accident.

Editing the ConfigMap is the only action required. The DRA driver watches the mounted policy file and reloads it in place when the ConfigMap changes, with no DaemonSet restart. Each update is validated before it takes effect; the driver rejects an invalid policy and keeps its last-good policy, so a bad edit does not take a node’s health monitoring offline. Confirm the reload and any rejected entries in the driver log:

kubectl logs -n neuron-dra-driver <driver-pod> | grep "taint policy"

Set manual device taints (DeviceTaintRule)

In addition to health-based taints set by the DRA driver, a cluster admin can taint Neuron devices manually with the Kubernetes DeviceTaintRule API. Use this for planned maintenance, fleet-wide cordoning, or manual overrides that are independent of health monitoring.

Note

The DeviceTaintRule API requires Kubernetes 1.37 or later — a higher floor than the rest of this feature, which needs Kubernetes 1.36.

A DeviceTaintRule adds a taint to devices that match a selector — by driver name, pool, device, or device class. For example, to cordon all Neuron devices on a node during maintenance:

apiVersion: resource.k8s.io/v1
kind: DeviceTaintRule
metadata:
  name: maintenance-node-1
spec:
  deviceSelector:
    driver: neuron.aws.com
    pool: ip-192-168-1-10.us-west-2.compute.internal
  taint:
    key: neuron.aws.com/maintenance
    value: scheduled
    effect: NoSchedule

The taint applies as long as the DeviceTaintRule exists, and you remove it by deleting the rule. The rule’s status reports how many devices and pods it affects without evicting anything.

For more information, see Taints set by an admin in the Kubernetes documentation.

Common issues

Health monitoring did not start

• Possible solution: Confirm the Neuron driver meets the version floor in Prerequisites and that the Neuron Observer DaemonSet is running. On an older driver, the per-device health signals are absent, so the DRA driver disables health monitoring and logs a warning. Check the driver log for that warning.

A device stays tainted and does not recover

• Possible solution: The driver removes a taint on the first clean reading from Neuron Observer, but some faults do not clear. Uncorrectable memory errors, in particular, do not recover. For a NoExecute taint, recycle the node to return its devices to a healthy state.

Related information

• AWS Neuron Dynamic Resource Allocation (DRA) — device taints require the Neuron DRA driver.

• Neuron Helm chart — install and configure the DRA driver.

• Neuron UltraServer Operator — provisioning and lifecycle management for UltraServer workloads.