This document is relevant for: Inf1, Inf2, Trn1, Trn2, Trn3

System Profile

The Neuron System Profile show a system-level granularity of execution across instances and workers in your workload. This provides visibility into Neuron Runtime API calls and ML framework function calls (PyTorch or JAX) to help identify bottlenecks in distributed workloads. The Neuron Explorer UI provides system-level widgets for an extensible and customizable workflow.

System Trace Viewer

The System Trace Viewer provides an interactive timeline interface with time range selection, configurable event grouping, system event details on hover, and linking of hardware events to Device Trace Viewer widgets.

You can see events in the Neuron Runtime and correlate them with hardware execution events on the Neuron Devices.

You can also see the device memory (HBM) allocations for each Neuron device over time. Hovering over these memory usage events shows a breakdown by usage category.

Adding Widgets

The System Profile supports both System and Device widgets, enabling multi-profile analysis, for example comparing annotated device events across different devices.

To add a widget:

1. Click the Add Widget button to open the Add Widget modal.

2. Select a Device or System widget.

3. Click a widget tile to load it with the selected profile. Each tile is tagged with its supported profile type (system, device, or both).

To load multiple instances of the same widget type for different profiles, repeat the steps above and select a different profile each time.

After adding a widget, you can switch to a different profile by using the profile dropdown at the top of the widget.

Note

Adding duplicate widgets for the same profile is not currently supported.

Settings

The System Trace Viewer supports multiple grouping modes to organize events for different analysis perspectives. You can switch between the following grouping modes in the settings to focus your analysis on different aspects of system performance:

Table 2 Grouping Options

Grouping Option	Description	Example
CPU vs Device Grouping (Default)	Groups events by event source (CPU or Neuron device events)	Runtime events: i-0b1ea78ca2865fd32/PID:1765325/TID:0/neuron_rt, Hardware events: i-0b1ea78ca2865fd32/PID:1765325/Worker:0/neuron_hw
NeuronCore Grouping	Groups events by individual NeuronCore	i-0b1ea78ca2865fd32/NC:0, i-0b1ea78ca2865fd32/NC:1
Thread Grouping	Groups events by thread identifier	i-0b1ea78ca2865fd32/PID:1765325/TID:0
Process Grouping	Groups events by process identifier	i-0b1ea78ca2865fd32/PID:1765325
Instance Grouping	Groups all events by instance only	i-0b1ea78ca2865fd32

Event Details

Clicking on trace events in the timeline populates the Event Details widget with a list of properties for the system trace event.

Device Profile Linking

The System Trace Viewer links hardware events to the Device Trace Viewer, which renders the corresponding device traces.

Navigating from the System Trace Viewer to a Device Trace Viewer can be accomplished in two ways:

Open the Device Profile List Modal

To see a list of all device profiles captured during your workload:

1. Click the “Device Profiles List” button in the top right action bar of the System Trace Viewer to open a modal containing a list of device profiles

2. Select a Device Profile and click Submit to open the Device Trace Viewer with the selected device profile

Drill-down from Hardware Events

To drill-down from a hardware event to the Device Trace Viewer:

1. Find a hardware event such as nc_exec_running

2. Click on the hardware event

3. Wait for the Device Trace Viewer to open

This will open a new Device Trace Viewer with the selected device profile showing detailed hardware events. To learn about device profiles, see Device Profiles in Neuron Explorer.

This document is relevant for: Inf1, Inf2, Trn1, Trn2, Trn3