Basic Neuron Tools

Identifying Neuron Devices

To identify number of Neuron Devices in a given instance use the neuron-ls command.

$ neuron-ls
+--------+--------+--------+-----------+--------------+---------+---------+---------+
| NEURON | NEURON | NEURON | CONNECTED |     PCI      | RUNTIME | RUNTIME | RUNTIME |
| DEVICE | CORES  | MEMORY |  DEVICES  |     BDF      | ADDRESS |   PID   | VERSION |
+--------+--------+--------+-----------+--------------+---------+---------+---------+
| 0      | 4      | 8 GB   | 1         | 0000:00:1c.0 | NA      | 12410   | NA      |
| 1      | 4      | 8 GB   | 2, 0      | 0000:00:1d.0 | NA      | 12410   | NA      |
| 2      | 4      | 8 GB   | 3, 1      | 0000:00:1e.0 | NA      | 12410   | NA      |
| 3      | 4      | 8 GB   | 2         | 0000:00:1f.0 | NA      | 12410   | NA      |
+--------+--------+--------+-----------+--------------+---------+---------+---------+

The above output is taken from an Inf1.6xlarge instance.

  • NEURON DEVICE -> Logical ID assigned to the NeuronDevice. This id can be used when configuring multiple runtimes to use different NeuronDevices.

  • NEURON CORES -> Number of NeuronCores present in the NeuronDevice.

  • NEURON MEMORY -> Amount DRAM memory in NeuronDevice.

  • CONNECTED DEVICES -> Shows other NeuronDevices connected to this NeuronDevice.

  • PCI BDF -> PCI Bus Device Function (BDF) ID of the device.

  • RUNTIME ADDRESS -> Shows address of runtime process using this NeuronDevice.

  • RUNTIME PID -> Shows process id of runtime process using this NeuronDevice.

  • RUNTIME VERSION -> Shows version of runtime process using this NeuronDevice.

NeuronCore Groups

Multiple NeuronCores(NC) can be combined to form a NeuronCore Group (NCG). Neuron framework layer will automatically create a default NeuronCore Group. To view list of available NCGs the following command can be used.

$ neuron-cli list-ncg
Device count 4 NC count 16
Found 4 NCG's
+--------+----------+--------------------+----------------+
| NCG ID | NC COUNT | DEVICE START INDEX | NC START INDEX |
+--------+----------+--------------------+----------------+
|      1 |        2 |                  0 |              0 |
|      2 |        4 |                  0 |              2 |
|      3 |        3 |                  1 |              2 |
|      4 |        1 |                  2 |              1 |
+--------+----------+--------------------+----------------+

The above examples shows there are 4 NCGs created on the system with the following grouping NCG ID 1: Device0:(Core0, Core1) NCG ID 2: Device0:(Core2, Core3), Device1:(Core0, Core1) NCG ID 3: Device1:(Core2, Core3), Device2:(Core0) NCG ID 2: Device1:(Core1)

Listing Models

Multiple models can be loaded into a single NCG, but only one can be in READY state at any given moment. Inference can be performed only on models in the READY state.

The neuron-cli list-model command should be used to view all the models.

$ neuron-cli list-model
+----------------------------------------------+----------+--------------+---------------+-------+-------+----------------------+
|                     UUID                     | MODEL ID | MODEL STATUS | NEURON DEVICE |  NC   |  NC   |         NAME         |
|                                              |          |              |  START INDEX  | INDEX | COUNT |                      |
+----------------------------------------------+----------+--------------+---------------+-------+-------+----------------------+
| 63c43dd60b0411eaa9160288cac7f65c30           |    10011 | STANDBY      |             1 |     0 |     1 | test0_1_concat_multi |
| 63c43dd60b0411eaa9160288cac7f65c30330808637f |    10010 | STANDBY      |             0 |     0 |     1 | test0_1_concat_multi |
| 63c43dd60b0411eaa9160288cac7f65ce078         |    10009 | READY        |             1 |     1 |     1 | test0_1_concat_multi |
| 63c43dd60b0411eaa9160288cac7f65ca05f         |    10008 | READY        |             1 |     0 |     1 | test0_1_concat_multi |
| 6a9726                                       |    10007 | READY        |             0 |     2 |     2 | onv_h1_2tpb_cpu_2tpb |
| 529c31da0b0411ea95730288cac7f65cb03b0afc627f |    10006 | READY        |             0 |     0 |     0 | t-test0_5conv_h1_cpu |
+----------------------------------------------+----------+--------------+---------------+-------+-------+----------------------+
  • UUID -> UUID generated for this model during compile time.

  • MODEL ID -> Neuron Runtime identifier for this model.

  • MODEL STATUS -> READY = The model is loaded on to the NeuronDevice and active on the NeuronCore. (Inference can be done only on models with READY state) STANDBY = The model is loaded on to the NeuronDevice but another model is currently active on the NeuronCore. (A model switch is needed to start inference)

View Resource Usage

Each model loaded consumes a different amount of memory (host and device), NeuronCore and CPU usage. The neuron-top command can be used to view the memory and NeuronCore usage.

$ neuron-top
neuron-top - 2020-02-12 23:03:15
NN Models: 2 total, 2 running
Number of VNCs tracked: 16

0000:00:1c.0 Utilizations: Neuron core0 0.00%, Neuron core1 0.00%, Neuron core2 0.00%, Neuron core3 0.00%,
0000:00:1e.0 Utilizations: Neuron core0 0.00%, Neuron core1 0.00%, Neuron core2 0.00%, Neuron core3 0.00%,

Model ID   Model Name                                                      UUID                               Node ID   Subgraph   Exec. Unit       Host Mem   Device Mem   Neuron core %
10018      1.0.6801.0-/home/ubuntu/benchmarking/compiler_workdir/rn50      d12cf238420d11ea8e270afe835c0a32   3         0          0000:00:1e.0:0   33554816   135290880    0.00
10017      1.0.6801.0-/home/ubuntu/benchmarking/compiler_workdir/rn50      d12cf238420d11ea8e270afe835c0a32   3         0          0000:00:1c.0:0   33554816   135290880    0.00

In the above output:

  • Model ID -> Unique Identifier for models loaded in the NeuronDevice

  • Model Name -> Neuron Compiler Version-compiler work directory/User defined model name

  • Node ID -> For Internal use only

  • UUID -> Unique Id assigned by the Neuron Compiler for a Model

  • Exec. Unit -> BDF of Neuron Device followed by the Neuron Core ID, b:d:f.NC

  • Host Mem -> Host memory consumed by the Model in bytes

  • Device Mem -> NeuronDevice memory consumed by the Model in bytes

  • Neuron Core % -> Utilization % of the neuron core at sample time. If there are no active inferences this value will be 0.