This document is relevant for: Trn2, Trn3

Reference: Neuron Observer configuration

Use this page to look up the type, default, and purpose of Neuron Observer Helm chart values commonly configured during the public beta. All values live under observer.* in the shared chart’s values.yaml, including fields that the chart writes to mounted component configuration files.

Note

Neuron Observer is in public beta. APIs, metric names, and installation steps can change before general availability.

Overview

For installation examples that use the observer.* values in these tables, see How to install and configure Neuron Observer on Amazon EKS.

Daemon image and scheduling

Value	Type	Default	Purpose
observer.enabled	bool	false	Set to true to render any Observer resource.
observer.image.repository	string	public.ecr.aws/neuron/neuron-observer	Container image repository.
observer.image.tag	string	0.2.0	Container image tag.
observer.image.pullPolicy	string	IfNotPresent	Kubernetes image pull policy.
observer.daemon.nodeSelector	map	neuron.amazonaws.com/present: "true"	Schedules the daemon only on nodes labeled as containing Neuron devices.
observer.daemon.hostNetwork	bool	true	Runs the daemon in the host network namespace so it can read EFA/RDMA hardware counters. The chart mounts host /proc separately; it does not enable hostPID.
observer.daemon.tolerations	list	[{key: aws.amazon.com/neuron, operator: Exists, effect: NoSchedule}]	Toleration for the Neuron taint. Extend it if your nodes have extra taints.

Prometheus scraping

The daemon always serves Prometheus metrics from /metrics on its API port. This native pull endpoint does not require observer.daemon.config.export.enabled or an OpenTelemetry collector.

Value	Type	Default	Purpose
observer.monitoring.releaseLabel	string	monitoring	Value of the release label added to Prometheus Operator resources. It must match the selector used by your Prometheus installation. Set it to an empty string to omit the label.
observer.monitoring.additionalLabels	map	{}	Labels added to all Observer Prometheus Operator resources.
observer.monitoring.podMonitor.enabled	bool or null	null	Controls creation of a PodMonitor. By default, the chart creates one only when the Prometheus Operator CRD is available. Set explicitly to true or false to override detection.
observer.monitoring.podMonitor.interval	duration	30s	Prometheus scrape interval.
observer.monitoring.podMonitor.scrapeTimeout	duration or null	unset	Optional Prometheus scrape timeout.
observer.monitoring.podMonitor.releaseLabel	string or null	unset	Overrides observer.monitoring.releaseLabel for the PodMonitor.
observer.monitoring.podMonitor.additionalLabels	map	{}	Additional labels added only to the PodMonitor.

Grafana dashboards

Enable the dashboard bundle to create ConfigMap resources for the cluster overview, node diagnostics, and device health dashboards. A Grafana dashboard sidecar container discovers these resources.

Value	Type	Default	Purpose
observer.monitoring.grafanaDashboards.enabled	bool	false	Creates the three dashboard ConfigMap resources when set to true.
observer.monitoring.grafanaDashboards.namespace	string	monitoring	Namespace where the chart creates the dashboard ConfigMap resources. The namespace must already exist.
observer.monitoring.grafanaDashboards.label	string	grafana_dashboard	Label key used by the Grafana dashboard sidecar to discover the ConfigMap resources.
observer.monitoring.grafanaDashboards.labelValue	string	"1"	Value assigned to the dashboard discovery label.
observer.monitoring.grafanaDashboards.additionalLabels	map	{}	Additional labels added to each dashboard ConfigMap.
observer.monitoring.grafanaDashboards.annotations	map	{}	Annotations added to each dashboard ConfigMap.

Daemon runtime configuration

The chart mounts these settings at /etc/neuron-observer/config.yaml.

Value	Type	Default	Purpose
observer.daemon.config.collection.interval_secs	int	5	Seconds between hardware collection cycles. Must be greater than zero.
observer.daemon.config.collection.proc_path	string	/host-proc	Absolute path to the procfs tree used for host and process metrics. The chart mounts the node’s /proc at this path.
observer.daemon.config.collection.sources.device	bool	true	Reads runtime device statistics and ioctl data through /dev/neuronN.
observer.daemon.config.collection.sources.sysfs	bool	true	Reads Neuron health, device information, and ECC data from sysfs.
observer.daemon.config.collection.sources.system	bool	true	Reads the configured procfs tree for host CPU, memory, and process metrics.
observer.daemon.config.collection.sources.imds	bool	true	Queries IMDS for instance_id, instance_type, and other instance metadata.
observer.daemon.config.collection.sources.efa	bool	true	Reads EFA network counters from /sys/class/infiniband.
observer.daemon.config.collection.latency_stream.enabled	bool	true	Enables background sampling used to produce per-core latency percentile metrics.
observer.daemon.config.collection.latency_stream.poll_interval_us	int	50	Microseconds between latency-stream samples. Must be greater than zero.
observer.daemon.config.emulation.enabled	bool	false	Replays a built-in snapshot instead of collecting from hardware. Intended for testing and development.
observer.daemon.config.export.enabled	bool	false	Enables OTLP push from the daemon. If export.endpoint is unset, the exporter uses the standard OpenTelemetry endpoint environment variables or the SDK default. This setting is independent of the native Prometheus /metrics endpoint.
observer.daemon.config.export.interval_secs	int	5	Seconds between OTLP exports. Must be greater than zero.
observer.daemon.config.export.endpoint	string or null	unset	Exact OTLP/HTTP metrics endpoint. A non-empty value overrides the standard OpenTelemetry endpoint environment variables and must include /v1/metrics.
observer.daemon.config.export.mapping	map or null	unset	Metric export mapping. Omit this value to use the built-in default, specify preset: monitor-compat to use the compatibility preset, or specify path: <absolute-path> to replace the default with a custom mapping file.
observer.daemon.config.logging.level	string	info	Tracing filter directive. Use a level such as trace, debug, info, warn, error, or off, or use a target-specific EnvFilter directive.
observer.daemon.config.logging.format	string	json	json or text.
observer.daemon.config.daemon.listen_address	string	0.0.0.0	IPv4 or IPv6 address on which the daemon API listens. Use 0.0.0.0 to listen on all IPv4 interfaces or :: to listen on all IPv6 interfaces.
observer.daemon.config.daemon.port	int	7001	TCP port for the daemon API and native Prometheus metrics. Must be greater than zero.
observer.daemon.config.daemon.shutdown_timeout_secs	int	5	Grace period for in-flight requests after shutdown begins. Must be greater than zero.
observer.daemon.config.events.health_buffer_capacity	int	4096	Maximum number of fault events retained in memory for replay to reconnecting clients.
observer.daemon.config.events.heartbeat_interval_ms	int	1000	Milliseconds between heartbeat frames on live event-stream connections.

OpenTelemetry export

When daemon export is enabled, the chart creates a separate daemon-mode OpenTelemetry collector.

These values configure the daemon collector:

Value	Type	Default	Purpose
observer.otelDaemonCollector.pipelines.cloudwatch.enabled	bool	false	Pushes metrics to Amazon CloudWatch through the EMF exporter.
observer.otelDaemonCollector.pipelines.cloudwatch.namespace	string	NeuronObserver	CloudWatch metric namespace.
observer.otelDaemonCollector.pipelines.cloudwatch.logGroupName	string	/metrics/neuron-observer	CloudWatch Logs log group used by the EMF exporter.
observer.otelDaemonCollector.pipelines.cloudwatch.logStreamName	string	otel-metrics	CloudWatch Logs stream used by the EMF exporter.
observer.otelDaemonCollector.pipelines.cloudwatch.storageResolution	int	1	CloudWatch metric storage resolution in seconds.
observer.otelDaemonCollector.pipelines.cloudwatch.metricDeclarationsOverride	list or null	null	Replaces the chart’s default EMF metric declarations when set.
observer.otelDaemonCollector.pipelines.amp.enabled	bool	false	Pushes metrics to Amazon Managed Service for Prometheus (AMP) through Prometheus remote write.
observer.otelDaemonCollector.pipelines.amp.endpoint	string	""	AMP workspace .../api/v1/remote_write endpoint. Required when this pipeline is enabled.
observer.otelDaemonCollector.pipelines.otlp.enabled	bool	false	Forwards metrics to another OTLP HTTP endpoint.
observer.otelDaemonCollector.pipelines.otlp.endpoint	string	""	Destination OTLP HTTP endpoint. Required when this pipeline is enabled.
observer.otelDaemonCollector.awsRegion	string	""	AWS Region used by the CloudWatch and AMP exporters. Required when either pipeline is enabled.

Namespace

Value	Type	Default	Purpose
observer.namespace.create	bool	false	Creates the neuron-observer namespace when set to true. When this value is false (the default), create the namespace before installation or use Helm’s --create-namespace option.

Related reference

• Reference: Neuron Observer metrics and faults — the metrics and fault categories Neuron Observer reports.

Learn more

• How to install and configure Neuron Observer on Amazon EKS — install and configure Neuron Observer on Amazon EKS.

• Get started with Neuron Observer on Amazon EKS — a quickstart from installation to first metrics.

• Troubleshooting and FAQs: Neuron Observer — common issues, FAQs, and beta limitations.

This document is relevant for: Trn2, Trn3