This document is relevant for: Inf2, Trn1, Trn2, Trn3

Neuron Runtime Async APIs: Core Concepts#

This page explains how the Neuron Runtime (NRT) explicit async APIs (nrta_*, declared in nrt_async.h) behave: how requests are queued, how completion is tracked, how errors are reported, and what you must synchronize yourself. Read it before writing an application against these APIs.

For motivation and background, see Neuron Runtime Async APIs: Motivation and Overview. For recommended usage patterns, see Best Practices and Async API Usage Examples. For function signatures, see nrt_async.h.

Summary#

  • Work is submitted to an execution unit (XU) queue. Each XU runs its requests one at a time, in submit order.

  • Every accepted request gets a sequence ID. You track completion by sequence ID, by polling or with an eventfd.

  • Scheduling calls do not block. If a queue is full they return NRT_QUEUE_FULL and change nothing.

  • A request’s final status is written to the ret pointer you pass at schedule time, before the request is reported complete.

  • Dependencies are either explicit or implicit. Requests on the same XU queue are implicitly ordered, so no synchronization is needed. For data produced on a different XU or queue, you must explicitly wait for the producer to complete before scheduling its consumer.

  • A fatal error permanently degrades an XU. Plan for recovery by restarting the application.

Execution units and queues#

Async requests are submitted to an execution unit (XU), an independent processing block of a Neuron Core (see Neuron Runtime Async APIs: Motivation and Overview). An XU is identified by (logical Neuron Core, XU type, queue). The lnc argument of the nrta_* APIs is the logical Neuron Core index:

XU

nrta_xu_t

Queues per logical core

Tensor Op

NRTA_XU_TENSOR_OP

2

Compute

NRTA_XU_COMPUTE

1

Collectives

NRTA_XU_COLLECTIVES

1

Each XU queue has its own sequence number space and health status. Requests on different XUs run in parallel; this is what lets you overlap data movement, compute and communication.

Sequence IDs and completion behavior#

Every request accepted by a schedule API is assigned a sequence ID, returned through a nrta_seq_t output argument. A sequence ID is a 64-bit value. The upper 16 bits identify the XU the request was submitted to, and the lower 48 bits are a counter that is specific to that XU:

 63            48 47                             0
+----------------+--------------------------------+
|   XU id (16)   |     sequence number (48)       |
+----------------+--------------------------------+

Because every XU keeps its own counter, sequence numbers are not unique across XUs on their own; the XU id in the upper bits is what makes the full 64-bit value unique. This also means sequence IDs should only be compared with other sequence IDs from the same XU.

Sequence numbers increase monotonically in submit order. When several threads submit to the same XU, the order is the order in which the runtime accepted the calls, so an application that needs a specific order across threads must serialize its submissions.

Requests on an XU also complete in the order they were submitted. This is what makes sequence IDs useful for tracking completion: the runtime only needs to keep track of the most recently completed sequence ID for each XU, and every request with a lower or equal ID on that XU is complete.

Use these APIs to check progress:

  • nrta_get_sequence() returns the most recently completed sequence ID for an XU.

  • nrta_is_completed(seq, &done) reports whether a specific request has completed.

  • Completion of sequence N implies completion of every earlier sequence on the same XU, so waiting on the last request of a batch is enough to know the whole batch is done.

A request that fails during execution still completes. Its final status is written to the NRT_STATUS *ret passed at schedule time before the request is reported as completed, so the status is safe to read as soon as completion is observed. Completion therefore means “the runtime is finished with this request”, not “the request succeeded”; check the returned status to tell the two apart.

The following sketch shows the pattern: schedule, wait, then read the status.

NRT_STATUS status = NRT_SUCCESS;   // must stay valid until the request completes
nrta_seq_t seq = {};

NRT_STATUS rc = nrta_execute_schedule(model, inputs, outputs, 0, &status, &seq);
if (rc != NRT_SUCCESS) {
    // Not accepted (for example NRT_QUEUE_FULL): no sequence was assigned.
}

bool done = false;
while (!done) {
    nrta_is_completed(seq, &done);   // or wait on an eventfd instead of spinning
}
if (status != NRT_SUCCESS) {
    // Accepted but failed during execution.
}

See Async API Usage Examples for complete examples and nrt_async.h for exact signatures.

Queue full behavior#

Every schedule API returns NRT_QUEUE_FULL when the target XU cannot accept another request. This is a clean failure: nothing was enqueued, no sequence was consumed, and the ret and req_sequence outputs are not modified. Retry once a request on that XU has completed. Rather than sleep-looping, wait on an XU completion event (see Event FD notifications) or poll nrta_get_sequence().

  • Compute XU: the limit is the number of requests that are scheduled and not yet completed, including the one currently running. The default is 8 and the maximum is 63; it is set with NEURON_RT_XU_COMPUTE_MAX_QUEUED_REQUESTS.

  • Tensor Op XU: each queue accepts a large number of outstanding requests (64K).

  • Collectives XU: nrta_cc_schedule can also return NRT_QUEUE_FULL; handle it the same way as for other XUs.

Request error behavior#

Errors are reported in one of two places.

  1. At schedule time, through the return value. The request was not accepted and no sequence was assigned. Typical statuses: NRT_INVALID (NULL arguments, bad core or queue, mismatched tensors), NRT_QUEUE_FULL, NRT_RESOURCE, NRT_CLOSED (the runtime is closing), NRT_EXEC_UNIT_UNRECOVERABLE (the XU is degraded).

  2. At execution time, through *ret. The request was accepted and has a sequence. When it completes, its final status is written through the NRT_STATUS *ret given at schedule time. ret may be NULL, but then a failure is not observable.

Rules to follow:

  • Keep ret valid until the request completes. The runtime writes *ret asynchronously while the request is in flight.

  • A failed request does not cancel the ones behind it. Each queued request completes with its own status.

  • Non-fatal and fatal failures. Invalid parameters, resource exhaustion and driver queue-full failures are non-fatal: the XU stays usable. Timeouts, hardware errors and unrecoverable driver errors are fatal and degrade the XU. Fatal failures are propagated to all other enqueued requests.

Data dependencies#

Between XUs/XU Queues: managed by the application#

The runtime does not track data dependencies between XU queues, and the schedule calls take no dependency arguments. If a request consumes data produced by a request on a different XU or queue, wait for the producer’s sequence to complete before scheduling the consumer. For example, an nrta_tensor_write that feeds an nrta_execute_schedule must be observed complete first. The same holds for execution → read, collective → execution, and so on. Use nrta_is_completed(), nrta_get_sequence() or an event fd to wait.

Scheduling a consumer early and hoping the producer finishes first is a race: the request can begin executing as soon as it reaches the head of its queue.

Within one XU queue: ordering is implicit#

Requests on the same XU queue run one at a time in submit order, so a consumer submitted after its producer on the same queue needs no explicit wait.

  • Compute: there is one queue per logical core, so executions that consume each other’s outputs need no synchronization.

  • Tensor Op: reads, writes, copies, casts, compute and FMA requests on the same queue are ordered with each other.

Event FD notifications#

Instead of polling, applications can register an eventfd file descriptor that the runtime signals by writing 1 to it (as an eventfd write) when work completes.

API

Signaled when

Lifetime

nrta_event_register_seq_id_completion(seq, fd)

The given sequence completes

One notification for that sequence

nrta_event_register_xu_completion(lnc, xu, queue, fd)

Any request on that XU completes

Until deregistered

Behavior:

  • The runtime does not own the fd. It never closes it. The fd must stay valid until it has been signaled or deregistered. An invalid fd is rejected with NRT_INVALID.

  • The fd must not block on write. Use eventfd(0, EFD_NONBLOCK). A blocking fd whose counter is saturated could stall completion processing for the affected core. If a non-blocking write fails with EAGAIN, that notification is lost.

  • Registering replaces. Registering again for the same sequence, or for the same XU, replaces the previous fd. One fd exists per sequence and per XU; fan out in the application if several consumers need the signal.

  • A negative fd deregisters. Deregistering an already completed sequence succeeds and does nothing.

  • Registering on an already completed sequence signals immediately, from the calling thread. This makes it safe to register right after scheduling, with no window in which a completion can be missed.

  • Sequences that were never issued return NRT_INVALID.

  • Ordering. The notification is sent after the completion is visible and after *ret has been written. On wake-up, nrta_is_completed(seq) is true and *ret holds the final status.

  • XU-level fds count completions. The XU fd is written once per completed request, so a burst of N completions adds N to the eventfd counter, and a single read returns the total. After waking, call nrta_get_sequence() to see how far the XU has progressed instead of assuming one completion per wake-up.

  • Degraded XUs still notify. Requests that complete with an error on a degraded XU still signal their fds.

Tip

Reuse a small set of eventfds rather than creating one per request.

Degraded XUs#

An XU that hits a fatal error becomes degraded. Degradation is permanent for that XU.

State

Meaning

Healthy

Normal operation

Closing

The runtime is being closed. New requests fail with NRT_CLOSED; requests already queued finish

Degraded

A fatal error occurred. New requests fail with NRT_EXEC_UNIT_UNRECOVERABLE

What degrades an XU

  • Compute: a fatal error while executing a model (timeout, HW error, and SW Queue Overflow). Problems detected before anything reaches the hardware (invalid arguments, resource preparation failures) fail only that call and do not degrade the XU.

  • Tensor Op: a request that completes with a fatal status (timeout, hardware error, unrecoverable driver error), or a partially posted request.

What happens once an XU is degraded

  • New schedule calls fail immediately with NRT_EXEC_UNIT_UNRECOVERABLE.

  • Requests that were already queued still complete, so their sequences advance, their event fds are signaled and their queue slots are released:

    • Compute: they are skipped and complete with NRT_EXEC_UNIT_UNRECOVERABLE. The request that hit the fatal error keeps its original error status.

    • Tensor Op: the requests were already handed to the hardware and finish with whatever status the driver reports.

  • A degraded XU cannot be recovered without at least re-initializing the runtime, and in the worst cases reloading the driver or rebooting the instance. When you see NRT_EXEC_UNIT_UNRECOVERABLE:

    1. drain the remaining in-flight work

    2. release resources

    3. notify upstream services

    4. restart the application and/or reload the driver

Scope. Health is tracked per XU. A degraded Tensor Op queue does not by itself mark the other Tensor Op queue or the Compute XU on that core as degraded, but an underlying hardware fault often causes failures elsewhere soon after, so check every XU’s results independently.

Threading#

  • Any thread may submit to an XU. Sequence order across threads is the order in which calls are accepted; the application is responsible for any ordering it needs between threads.

  • The runtime completes requests and sends event fd notifications from its own internal threads, not from application threads. Do not assume a notification arrives on a particular thread.