This document is relevant for: Inf2, Trn1, Trn2, Trn3

nrtpy troubleshooting#

This page lists common issues encountered when using nrtpy and their solutions.

Import and setup issues#

import nrtpy fails with “libnrt.so.1: cannot open shared object file”#

The Neuron Runtime library is not on the library path. Set it before running Python:

export LD_LIBRARY_PATH=/opt/aws/neuron/lib:$LD_LIBRARY_PATH

Verify the library exists:

ls /opt/aws/neuron/lib/libnrt.so*

If missing, install aws-neuronx-runtime-lib.

Import conflict between nrtpy and nki#

The standalone nrtpy wheel and the nki wheel cannot be installed in the same Python environment due to a namespace conflict. In a future release, this limitation will be resolved. Until then, install only one per environment:

  • Use your NKI environment to compile kernels to NEFFs.

  • Use a separate nrtpy environment to load and execute NEFFs.

Model loading issues#

NrtError: NRT_FAILURE(6) on load_from_neff#

The NEFF was compiled for a different target architecture than the current instance. Recompile with the correct --target (for example, trn2 for Trn2 instances).

NEFF file not found after compilation#

When using NKI_ARTIFACTS_DIR, the NEFF is saved as kernel.neff in the specified directory. When using torch_neuronx.trace with compiler_workdir, the NEFF filename includes a hash. Use:

find <artifacts_dir> -name "*.neff"

NKI_ARTIFACTS_DIR must be set before importing nki. If set after, the compile options will not pick it up and the NEFF goes to a temp directory that is deleted after execution.

Execution issues#

Input name mismatch error#

The dictionary keys passed to model() must exactly match the NEFF’s expected input names. Use model.input_tensors_info.keys() to discover the correct names:

print(model.input_tensors_info.keys())

Output tensor has void dtype (|V2, |V4)#

Auto-allocated output tensors may report a void dtype because libnrt does not always report the original dtype. Reinterpret the raw bytes with NumPy:

result = output_tensor.numpy().view(np.float16)   # for fp16 kernels
result = output_tensor.numpy().view(np.float32)   # for fp32 kernels

NrtError: NRT_EXEC_OOB(1006)#

The kernel attempted an out-of-bounds memory access at runtime. This typically means DMA indices point outside the source or destination tensor bounds. Check your kernel’s indirect addressing logic.

NRT runtime also emits diagnostic lines to stderr (TDRV, NMGR, NRT prefixed) before the Python exception is raised. These provide the affected NeuronCore and model path.

Benchmarking issues#

Benchmark variance is high (std_dev > 10% of mean)#

Increase warmup_iter to ensure the kernel is fully warmed up. Also check that no other workload is sharing the NeuronCore.

Device benchmark shows more iterations than benchmark_iter#

Some kernels emit multiple execution events per call (multi-subgraph NEFFs). The reported iterations in BenchmarkResult reflects actual device events, not Python calls.

Benchmark raises RuntimeError in device mode#

In device mode, the NeuronCore must support system tracing. Ensure your SDK version is 2.32 or later and that no other process is holding the NeuronCore.

Trace file is empty or very small#

Ensure save_trace=True is passed and that execution completes without error. Traces are only written on successful execution.

This document is relevant for: Inf2, Trn1, Trn2, Trn3