.. meta:: :description: Complete release notes for the Neuron Compiler component across all AWS Neuron SDK versions. :keywords: neuron compiler, neuronx-cc, release notes, aws neuron sdk :date-modified: 10/19/2026 .. _compiler_rn: Component Release Notes for Neuron Graph Compiler ================================================== **Latest version (in 2.33.0)**: 2.28.7631.0 The release notes for the Neuron Graph Compiler (``neuronx-cc``) Neuron component. Read them for the details about the changes, improvements, and bug fixes for all release versions of the AWS Neuron SDK. .. note:: For older Neuron Compiler (neuron-cc) release notes, see :doc:`the archived Neuron Compiler release notes ` and :doc:`Neuron Compiler operations release notes `. .. _compiler-2-33-0-rn: Neuron Compiler [2.28.7631.0] (Neuron 2.33.0 Release) ----------------------------------------------------- Date of Release: 10/19/2026 Changes ~~~~~~~ * **Faster compilation for models captured through torch.compile**: Compile time is substantially reduced for ``torch.compile`` models with heterogeneous architectures, such as vision-language, encoder-decoder, and mixture-of-experts models. For example, compile time for a large MoE backward graph improved by more than 20x in Neuron internal benchmarks. No changes to user code or compiler flags are required. * **LNC consistency check for NKI kernels enabled by default**: The compiler now verifies that every NKI kernel in a graph is configured for the same logical NeuronCore (LNC) count as the compiler's ``--lnc`` setting, and fails with :doc:`NCC_EVRF066 ` on a mismatch. Previously, a mismatch could compile successfully but produce incorrect results. To restore the previous behavior, set the environment variable ``NEURON_ENFORCE_NKI_LNC_CONSISTENCY=0``. * **Integer narrowing conversions now wrap instead of saturating**: Conversions from a wider integer type to a narrower one (for example, ``int32`` to ``int8``) now truncate using two's-complement wrap-around, matching the behavior of StableHLO, PyTorch, and CPU execution. Previously, out-of-range values were saturated to the minimum or maximum of the destination type. * **Clearer errors for invalid input HLO**: Input graphs that fail HLO shape or structural verification now report :doc:`NCC_EVRF064 ` and identify the offending instruction, instead of producing an internal compiler error. * **64-bit integer flag defaults unchanged (default change postponed)**: The default values of ``--native-int64`` (``disabled``) and ``--implicit-integer-downcast`` (``all``) introduced in the previous release remain unchanged in this release. The planned default change has been postponed: **in the next SDK release, native int64 execution will be enabled by default (--native-int64=enabled and --implicit-integer-downcast=none).** Users who depend on the current downcast behavior should pin --native-int64=disabled and --implicit-integer-downcast=all explicitly. This release includes additional correctness fixes for ``--native-int64=enabled``. Performance and Compile Time Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * **Faster MXFP8 models on trn3**: Quantized weights and MX matrix multiplication outputs now stay in on-chip memory under LNC=2 sharding instead of being routed through HBM, reducing latency by about 7x on an MXFP8 training workload. For MXFP8 models that take externally supplied scale tensors, scales are now laid out for wide, contiguous prefetch, reducing latency by up to 30x on affected workloads, without changing the expected input layout. * **Large backend compile time reductions**: Several backend analyses have been rewritten to scale better on large graphs. Instruction scheduling on graphs with many dynamic loops is up to 67x faster on a large MoE decode graph, loop-carried dependency analysis is up to 72x faster on very large graphs, and memory lifetime analysis for graphs with many basic blocks is up to 9x faster. Faster anti-dependency analysis alone reduces aggregate compile time by about 12% across Neuron internal benchmarks (up to 22% on individual models), and scheduler setup, DMA lowering, and loop unrolling are each 40-65% faster on large language models. * **Faster compilation for graphs with aliased inputs and outputs**: Alias analysis results are now cached correctly, reducing graph compiler time by about 80x on an affected video-generation model. * **Faster compilation for models with custom kernels**: Built-in kernel binaries are now reused across subgraphs, and kernel hashing is skipped when it is not needed, reducing the time spent on these phases by about 50-60% on large language models. * **Reduced memory usage for integer scatter with unique indices**: Integer scatter operations with unique indices (for example, ``uint8`` or FP8 KV-cache updates) no longer create a full-size FP32 copy of the destination tensor. This sharply reduces peak device memory and enables longer context lengths with FP8 KV caches. * **Reduced memory usage for batched linear layers**: Batched matrix multiplications whose weight operand is only broadcast across the batch dimension (common for ``nn.Linear`` applied to 3-D or higher-rank activations) are now converted to a single 2-D matrix multiplication, avoiding large replicated weight copies at large batch sizes. * **More efficient slices, concatenations, and contiguous gathers**: Constant-offset slices that narrow the innermost dimension, and contiguous concatenations of inputs or constants, are now lowered to direct DRAM-to-DRAM copies. Gather and scatter operations indexed by a contiguous range known at compile time are lowered to slice and concatenate operations. For innermost-dimension slices, this reduces NEFF size by up to 375x and improves runtime performance by up to 5x on affected models. * **Fewer DMA operations for strided and high-rank tensor accesses**: Strided DMA vectorization is now enabled by default, merging large numbers of small DMAs into a single scatter DMA (for example, 524,288 small DMAs into one). Strided accesses on tensors with 8 or more dimensions now use a supported transpose, reducing DMA count by up to 3x on affected models. Bug Fixes ~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 80 * - Area - Fix * - Compilation error - Fixed out-of-memory compilation failures for cumulative sums over broadcast tensors, such as those generated by ``masked_scatter`` and its backward pass, which previously required more than 24 GB of intermediate memory (:doc:`NCC_EOOM002 `, :doc:`NCC_EVRF007 `) * - Compilation error - Fixed verifier failures for scatter operations whose update values have a different floating-point type than the destination tensor, such as ``index_add`` with FP32 updates into a BF16 tensor (NCC_IVRF100) * - Compilation error - Fixed several compilation failures for MXFP8 models on ``trn3``, including quantization following RMSNorm (NCC_IISA901), quantization in scaled matrix multiplication training graphs (NCC_ILTO901), mismatched data and scale store layouts (NCC_IRAC902), row-tiled MX matrix multiplications (NCC_IGCA043), E8M0 scale transposes (NCC_IBCG901), and factorized MX layouts * - Compilation error - Fixed on-chip memory allocation failures on large MoE training graphs, fragmented state buffers, and loops that read the same tensor multiple times concurrently (NCC_IGCA030, NCC_IGCA113, NCC_IBIR228, NCC_IBIR229) * - Compilation error - Fixed a crash during state buffer spill insertion when a memory location is read before it is first written (for example, Mamba models compiled with ``torch.compile``) * - Compilation error - Fixed an ISA validation failure for TopK on ``trn3`` by placing its value and index outputs on the same base partition (NCC_IINAR001) * - Compilation error - Fixed an internal error for TopK over large, unevenly divisible axes (NCC_IXCG856) * - Compilation error - Fixed a topological order violation when DMA prefetching reordered writes ahead of reads on reused buffers (NCC_ISCH719) * - Compilation error - Fixed compilation failures for transposed convolutions compiled with LNC=2 (NCC_ILSM901) * - Compilation error - Fixed a type legalization crash when a collective's producer operation was folded into the collective, affecting large MoE decoders at LNC=2 (NCC_ILTY902) * - Compilation error - Fixed a false verifier error on collectives with 2\ :sup:`32` or more elements caused by a 32-bit overflow (NCC_INLA001) * - Compilation error - Fixed an out-of-bounds memory error where partition-axis tiling produced state buffer tiles with more than 128 partitions (NCC_INLA001) * - Compilation error - Fixed an out-of-bounds access pattern error in depthwise convolution when the DMA tile size is not a multiple of the compute tile size (NCC_IBIR243) * - Compilation error - Fixed crashes in loop fusion, dead store elimination, broadcast store conflict resolution, tensor initialization, mask propagation, and partition vectorization (NCC_ILFU901, NCC_IDSE902, NCC_IIOT902, NCC_ITIN902, NCC_IMPR902) * - Compilation error - Fixed compilation failures after folding back-to-back transposes and for integer transposes after copy folding (NCC_ILLP901, NCC_IBCG901) * - Compilation error - Fixed a crash when activation operations with a partial sum buffer input execute over multiple loop iterations * - Compilation error - Fixed a compiler assertion when a compute instruction's free-dimension step exceeds the range supported by hardware * - Compilation error - Fixed false verifier errors on empty tensor slices and zero-element access patterns * - Compilation error - Fixed intermittent ``neuronx-cc terminated abnormally`` failures at process exit after a successful compilation, most often seen when running many compilations in parallel * - Compilation error - Fixed compilation failures when ISLpy 2026.2 is installed (NCC_ISMP902) * - Compilation error - Fixed ``neuronx-cc --help`` failing to display * - Compiler bug - Fixed NaN outputs in masked softmax (for example, scaled dot-product attention with fully masked rows) caused by incorrect initialization of tiled max and min reductions * - Compiler bug - Fixed FP32 sum reductions returning NaN instead of ±infinity when the input contains infinite values * - Compiler bug - Fixed incorrect results from product reductions (for example, ``torch.prod``) over broadcast tensors * - Compiler bug - Fixed incorrect results from reverse arithmetic right shifts (scalar shifted by a tensor) and from right shifts folded into memory initialization * - Compiler bug - Fixed in-place copies within an aliased input/output buffer (for example, KV-cache shifts) being incorrectly removed, and fixed stale reads of aliased inputs accessed through reshape, transpose, or convert after an in-place update * - Compiler bug - Fixed non-deterministic results in multi-subgraph models with collective operations caused by collective handshakes being skipped incorrectly * - Compiler bug - Fixed silently truncated results for switch-accelerated collectives on ``trn3`` by aligning collective output buffers of 64 MB or smaller * - Compiler bug - Fixed a data race where a register load could execute before an in-flight DMA completed, causing incorrect results in models that use indirect DMA offsets * - Compiler bug - Fixed missing loop-carried dependencies in dynamic loops that reuse the same state buffer memory for different tensors * - Compiler bug - Fixed incorrect results from operations placed on engines that cannot execute them correctly, including certain tensor-scalar and mixed integer/floating-point operations * - Compiler bug - Fixed incorrect results when sharding a scatter whose dimension is also a softmax reduction dimension * - Compiler bug - Fixed incorrect MXFP8 quantization output ordering with factorized MX layouts on ``trn3`` * - Compiler bug - Fixed potential stale reads when loads of a tensor surround an in-place write and share a transpose * - Compiler bug - Fixed dynamic-offset DMAs into DRAM tensors larger than 4 GiB * Other minor bug fixes and performance enhancements for ``trn1``, ``trn2``, and ``trn3`` platforms. .. _compiler-2-32-0-rn: Neuron Compiler [2.27.5334.0] (Neuron 2.32.0 Release) ------------------------------------------------------ Date of Release: 08/17/2026 Changes ~~~~~~~ * **New public flags for 64-bit integer control (--native-int64 and --implicit-integer-downcast)**: Two new flags give users explicit control over 64-bit integer compilation behavior. ``--native-int64=enabled`` runs int64 ops natively without precision loss on ``trn2`` and later; ``--implicit-integer-downcast`` controls which ops with no native integer datapath (dot, scatter, all_reduce, reduce_scatter, pow) are silently downcast versus hard-errored. The default posture preserves backward compatibility: int64 ops are downcast to int32, and unsupported ops downcast with a warning. **In the next SDK release, the defaults will change to --native-int64=enabled and --implicit-integer-downcast=none, which enables native int64 execution and hard-errors on unsupported ops. Users who depend on the current lossy downcast behavior should pin --native-int64=disabled and --implicit-integer-downcast=all explicitly.** * **Expanded complex64 data type support**: Additional complex64 operations are now supported, bringing the total to 30 operations including: add, subtract, multiply, divide, compare, abs, exponential, sine, cosine, reshape, complex, real, imag, dot_general, broadcast_in_dim, iota, negate, power, log, log_plus_one, exponential_minus_one, sqrt, rsqrt, tan, tanh, logistic, sign, select, transpose, reverse, slice, dynamic_slice, dynamic_update_slice, concatenate, pad, and dot. Models using complex arithmetic through these operations can now compile without manual decomposition. Performance and Compile Time Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * **One-hot embedding lookup optimization**: The compiler now automatically rewrites one-hot matmul embedding lookups (``dot(one_hot(indices), table)``) into equivalent gather operations. This eliminates the materialized one-hot intermediate tensor (often ~256 MB) and replaces an expensive matmul with a single gather, reducing compile time by up to 64% and NEFF size by up to 96% for affected patterns. * **Factorized MX scale layout support on trn3**: The compiler now supports factorized scale layouts for MXFP8 quantized models on ``trn3``, enabling more efficient memory access patterns for mixed-precision training workloads. * **Reduced weight load frequency for MXFP8 matrix multiplication on trn3**: When consecutive matrix multiplications share the same stationary weight operand (common in MoE and multi-head architectures), the compiler now reuses the loaded weight and scale data instead of re-fetching from HBM for each operation. On affected models this eliminates redundant weight loads (e.g., 1 load instead of 128 for a shared-weight sequence), reducing HBM bandwidth pressure and improving throughput for MXFP8 quantized workloads on ``trn3``. Bug Fixes ~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 80 * - Area - Fix * - Compilation error - Fixed a crash in loop predication when comparing shard indices outside the loop nest (NCC_IRPX902) * - Compilation error - Fixed a LoopFusion crash when candidate fusions would create imperfect loop nests with conflicting schedules * - Compilation error - Fixed a crash when lowering scatter operations with scalar operands * - Compilation error - Fixed a crash in quantized model compilation when shared MXFP8 activations were lowered * - Compilation error - Fixed a DMA engine assignment crash for unassigned copy operations * - Compilation error - Fixed a compilation failure where on-chip memory live-range hints were not cleared between allocation passes, causing spurious conflicts * - Compilation error - Fixed a compilation failure for convolution operations where output buffer overlap occurred with multiple tile configurations * - Compilation error - Fixed a linker failure for LNC=1 configurations with certain subgraph layout patterns * - Compiler bug - Fixed incorrect results from the Simplifier pass when a multiply-reduce operation was broadcast across dimensions * - Compiler bug - Fixed incorrect partition vectorization that corrupted transpose annotations, leading to wrong output values * - Compiler bug - Fixed incorrect results when 64-bit to 32-bit integer casts were incorrectly folded into DMA operations * Other minor bug fixes and performance enhancements for ``trn1``, ``trn2``, and ``trn3`` platforms. .. _compiler-2-31-0-rn: Neuron Compiler (Neuron 2.31.0 Release) ---------------------------------------------------- Date of Release: 07/07/2026 Changes, Performance and Compile Time Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * **StableHLO composite operation mapping**: The compiler can now map StableHLO composite operations (including attention kernels) directly, enabling more efficient compilation of models expressed using the StableHLO composite format. * **New code generation backend enabled by default on trn2 and trn3**: A redesigned instruction code generation backend is now the default for ``trn2`` and ``trn3`` targets. * **LNC-aware instruction scheduling**: The post-scheduler now accounts for Logical NeuronCore (LNC) constraints when ordering instructions, reducing synchronization overhead and improving runtime performance for LNC=2 workloads. * **Pre-scheduling DMA prefetch**: The compiler now inserts DMA prefetch hints during the pre-scheduling phase, allowing memory transfers to overlap with computation earlier in the execution pipeline. This reduces memory stall cycles on models with large weight tensors. * **Faster dependency analysis (ADA optimization)**: The anti-dependency analyzer now skips trailing read operations that have no subsequent write, eliminating unnecessary dependency edges. This reduces compile time for models with many read-heavy memory access patterns. * **Improved compilation for heterogeneous models**: The compiler now performs fuzzy re-splitting of computation tiles for models with heterogeneous layer structures, producing more balanced workload distribution that reduces peak memory usage and compile time. Bug Fixes ~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 80 * - Area - Fix * - Compilation error - Fixed a crash in tensor initialization when processing non-shrink memset paths * - Compilation error - Fixed incorrect collective operation rank-0 semantics for permute and reduce-scatter, which could cause compilation failures or hangs on multi-node models * - Compilation error - Fixed a vector accumulation register collision caused by StreamTranspose cross-bank over-read during instruction scheduling * - Compilation error - Fixed incorrect bank address computation for preallocated tensors starting at non-aligned offsets when using vector accumulation registers, which could cause tensor overlap * - Compilation error - Fixed a segfault in DMA prefetch when rolling back performance simulation state * - Compilation error - Fixed scheduling quality regression on workloads with heavy synchronization * - Compilation error - Fixed non-deterministic compilation failures caused by data races in control flow validation, constant propagation ordering, and internal optimization heuristics * - Compilation error - Fixed a loop fusion crash when candidates would create imperfect loop nests * - Compilation error - Fixed a tensor initialization crash where memset was inserted at incorrect program points * - Compiler bug - Fixed incorrect tensor reordering in the simplifier that could produce non-deterministic optimization results * - Compiler bug - Fixed negative-stride copy fusion that incorrectly merged reverse operations into compute * Other minor bug fixes and performance enhancements for ``trn2`` and ``trn3`` platforms. .. _compiler-2-30-0-rn: Neuron Compiler (Neuron 2.30.0 Release) ---------------------------------------------------- Date of Release: 05/21/2026 Changes ~~~~~~~ * **Automatic boolean addition to OR conversion**: When a model uses addition on boolean tensors, the compiler now automatically treats it as a logical OR operation. This matches the behavior expected by popular ML frameworks and avoids unexpected numerical results. * **Coalesced ordered reduce-scatter support**: The compiler now recognizes grouped reduce-scatter operations (where multiple reduce-scatters execute in a defined order) and handles them as a single optimized unit, improving efficiency for distributed training communication patterns. Performance and Compile Time Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * **Faster compilation for large kernels (liveness analysis)**: The compiler's memory liveness analysis has been rewritten from an iterative fixpoint algorithm (which processed all tensors across all basic blocks simultaneously) to a per-tensor backward traversal that only visits blocks where each tensor is actually live. Tensors defined and used within a single basic block — common in NKI kernels — require zero traversal. This reduces the on-chip memory allocation phase from ~1,586s to ~42s (38×) on the most affected customer model, with a mean -1.4% total compile time improvement across Neuron internal benchmarks. * **Smarter memory analysis for strided tensors**: The compiler now applies conservative memory analysis only to tensors that actually need it (those exceeding 100K access intervals), rather than applying it globally. High-interval tensors are coarsened to 4KB-aligned intervals. This preserves optimization quality for most tensors while preventing excessive compile time on heavily-strided tensors. * **Faster dependency building for high-fanout operations**: Linearize dependencies for high-fanout memory locations when overlap-check cost exceeds 1M checks, eliminating quadratic compile-time overhead. * **Larger computation tiles (up to 1024)**: The maximum tile size for computation has been increased from 512 to 1024, allowing the compiler to choose larger tiles that reduce loop overhead and improve runtime performance for large tensor operations. * **Improved memory access patterns in load/compute separation**: The compiler now generates more efficient memory initialization patterns when separating load and compute phases, improving DMA performance. * **Better tensor layout selection for concatenated outputs**: The compiler now avoids placing concatenation dimensions in the innermost position, preventing inefficient single-element-per-partition DMA writes that previously degraded performance on certain models. * **Reduced memory pressure for matrix multiplication prefetch**: The compiler now avoids over-buffering matrix multiplication operands in certain tiling configurations, reducing on-chip memory pressure and improving allocation success. * **Improved handling of constant tensors**: The compiler can now cheaply regenerate constant values (zeros, sequential indices) on-chip instead of spilling them to DRAM, reducing memory traffic for kernels with many constant-initialized tensors. Additionally, repeated constant values larger than 16 bytes are now decomposed into scalar broadcasts, reducing NEFF file size and improving load-time DMA efficiency. Bug Fixes ~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 80 * - Area - Fix * - Compilation error - Fixed a failure where certain tensor configurations exceeded on-chip memory capacity during compilation (NCC_IBIR229) * - Compilation error - Fixed a crash when compiling softmax backward operations with certain input shape combinations (NCC_ILIN902) * - Compilation error - Fixed an uninitialized read error when combining collective operations that include NKI kernel outputs (NCC_IBIR182) * - Compilation error - Fixed several compiler crashes affecting tensor spill/reload, collective operation coalescing, metric serialization at -O1, and constant folding with type conversion * - Compilation error - Fixed an internal verification failure after tiling broadcast operations * - Compilation error - Fixed a rare data race that could cause non-deterministic compilation failures in control flow validation * - Compilation error - Fixed out-of-bounds errors and hangs on multi-rank models during load/compute separation * - Compilation error - Fixed hangs caused by missing dependencies when separating indirect memory loads from compute * - Compilation error - Fixed a compilation failure for models with large vocabulary dimensions where on-chip tensor placement could not proceed * - Compilation error - Fixed a compilation failure where address pattern validation did not re-check constraints after unrolling * - Compiler bug - Fixed incorrect results in select-and-scatter (windowed reduction) operations due to missing boundary conditions * - Compiler bug - Fixed off-by-one errors in integer division operations caused by insufficient precision in internal computation * - Compiler bug - Fixed incorrect convolution output when kernel_size is smaller than dilation, where some output positions were not properly zero-initialized * - Compiler bug - Fixed incorrect tensor layout selection that could produce wrong results for certain partition configurations * - Compiler bug - Fixed a hang in multi-chip collective operations on trn2 and trn3 by enabling DMA transpose chaining with collectives and remote memory accesses by default * Other minor bug fixes and performance enhancements for ``trn1``, ``trn2``, and ``trn3`` platforms. .. _compiler-2-29-0-rn: Neuron Compiler [2.24.5133.0] (Neuron 2.29.0 Release) ----------------------------------------------------- Date of Release: 04/09/2026 Changes ~~~~~~~ * **Gather bounds verification in HLO verifier**: The verifier now detects when gather operation iota indices exceed operand tensor bounds, providing a clear early error (:doc:`NCC_EVRF056 `) instead of confusing errors deep in compilation. This helps catch misconfigured models where ``max_position_embeddings < sequence_length``. * **Collective Broadcast to AllGather transformation**: The compiler automatically transforms collective broadcast operations into more efficient AllGather operations where applicable. * **DVE Exponential instruction support**: The compiler now supports exponential operations on the DVE engine. Performance and Compile Time Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * **Cost model for tensor-contract vs mul+reduce-add trade-off**: A cycle-based cost model decides between TensorContract (PE engine) and multiply-reduce_add for matrix multiplication patterns. Fixes sub-optimal tiling on batch matmul, reducing compile time from 20 seconds to 1 second in affected cases. * **Dependency reduction parallelized for large sequence lengths**: The dependency reduction phase now runs in parallel, yielding ~2x pass-time reduction in internal benchmarks for 64K sequence length models. * **DMA coalescing uses interval tree for O(log n) conflict detection**: ~20% median reduction in compile time improvements for DMA optimization passes across 408 Neuron internal benchmarks, with up to 50% reduction on the largest models. * **Improved hardware-aware tensor splitting**: DRAM splitting is now optional and the maximum unsplit tensor size is hardware-aware, reducing unnecessary splitting overhead. * **Improved index bounds checking for constant tensors**: Validation now checks against specific tensor dimension sizes and throws an ``NCC_ESMP002`` error if it is out of bound. Bug Fixes ~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 20 60 * - Area - Error Code - Fix * - Compilation failure - NCC_CFG015 - Fixed stack overflow in control flow graph validation for deeply nested models (e.g., flash-attention backward with 8193 blocks) * - Compilation error - NCC_IBIR229 - Fixed SimplifyTensor expanding SBUF tensors beyond per-partition capacity, causing downstream allocation failures * - Compilation error - NCC_INLA001 - Fixed legalization failure with >2 free access pattern dimensions, fixing 4 FalconH105b CrossEntropyLoss compilation failures * - Compiler bug - — - Fixed inverted matmul accumulate flags causing incorrect accumulation behavior * - Compiler bug - — - Fixed dead code elimination incorrectly removing writes to input tensors with no user * - Compiler bug - — - Fixed dead code elimination handling of must-alias tensors with no reader * - Compiler bug - — - Fixed ``-Inf`` being incorrectly replaced with constant in clamp operations * - Compilation error - — - Fixed broadcast store optimization when load has multiple users — cloned loop variables were referencing out-of-scope indices * - Compiler bug - — - Fixed SB partition address legalization for atomic load/store operations after post-split * Other minor bug fixes and performance enhancements for ``trn1``, ``trn2``, and ``trn3`` platforms. ---- .. _compiler-2-27-0-rn: Neuron Compiler [2.22.12471.0] (Neuron 2.27.0 Release) -------------------------------------------------------- Date of Release: 12/19/2025 Improvements ~~~~~~~~~~~~~~~ * New error code documentation has been added to help developers better understand and troubleshoot issues encountered during model compilation. * Two Neuron Compiler (neuronxcc) flags now have different default behaviors to improve accuracy. The ``--auto-cast`` flag now defaults to ``none`` (previously ``matmul``), and ``--enable-mixed-precision-accumulation`` is now enabled by default. Breaking Changes ~~~~~~~~~~~~~~~~ * Python 3.9 no longer supported: The Neuron Compiler requires Python 3.10 or higher. Users currently on Python 3.9 must upgrade to continue using the Neuron Compiler with Python bindings. * Compiler accuracy flag defaults updated: These changes optimize accuracy but may impact performance for FP32 models and models using smaller bitwidth dtypes. To restore previous behavior, explicitly set ``--auto-cast=matmult`` and use the new ``--disable-mixed-precision-accumulation`` flag. Bug Fixes ~~~~~~~~~ * Minor bug fixes and performance enhancements for both the ``trn1`` and ``trn2`` platforms. ---- .. _compiler-2-26-0-rn: Neuron Compiler [2.21.18209] (Neuron 2.26.0 Release) -------------------------------------------------------- Date of Release: 09/08/2025 Bug Fixes ~~~~~~~~~ * Minor bug fixes and performance enhancements for both the ``trn1`` and ``trn2`` platforms. ---- .. _compiler-2-25-0-rn: Neuron Compiler [2.20.9961.0] (Neuron 2.25.0 Release) ------------------------------------------------------ Date of Release: 07/31/2025 Breaking Changes ~~~~~~~~~~~~~~~~ * Announcement: The Neuron Compiler default for the ``--auto-cast`` option will change from ``--auto-cast=matmult`` to ``--auto-cast=none`` in a future release. Bug Fixes ~~~~~~~~~ * Minor bug fixes and performance enhancements for both the trn1 and trn2 platforms. Known Issues ~~~~~~~~~~~~ * The Llama3 70B test has a compile time increase of 16% and 18%, for 16 and 32 nodes respectively. ---- Neuron Compiler [2.19.8089.0] ------------------------------- Date: 06/24/2025 * This update enables the Hardware DMA Generation Engine (Hardware DGE) by default on Trainium2 instances. Hardware DGE is a memory optimization that will reduce the generated compiler artifacts size (i.e., NEFFs) and the models’ memory footprint. Data movement (DMA) descriptors will be generated in the hardware, as needed, during model execution. This reduces (HBM) memory usage within the NEFF and allows the use of more DMA queues. ---- Neuron Compiler [2.18.121.0] ---------------------------- Date: 05/19/2025 * Minor bug fixes and performance enhancements for both the trn1 and trn2 platforms. ---- Neuron Compiler [2.17.194.0] ---------------------------- Date: 04/03/2025 * Minor bug fixes and performance enhancements for both the trn1 and trn2 platforms. ---- Neuron Compiler [2.16.372.0] ---------------------------- Date: 01/14/2025 * Minor bug fixes and performance enhancements for the trn2 platform. ---- Neuron Compiler [2.16.345.0] ---------------------------- Date: 12/20/2024 * Minor bug fixes and performance enhancements for the trn2 platform. ---- Neuron Compiler [Neuron 2.21.0 Beta] -------------------------------------- Date: 12/03/2024 * This release introduces the ``trn2`` option argument to the compiler ``--target`` option to specify that the compiler should generate code for a trn2 instance family. Example usage: ``neuronx-cc compile --target=trn2 ...`` * This release introduces the ``--logical-nc-config`` or ``-lnc`` compiler command line option in support of the Logical NeuronCore Configuration feature available in Trainium2 instances. The compiler's default is LNC=2. **Note: Use of this option is available only for Trainium2 instances.** ---- Neuron Compiler [2.15.128.0] ---------------------------- Date: 09/16/2024 * This release introduces memory optimization that will reduce the generated compiler artifacts size (i.e., NEFFs) and the models' memory footprint. It is possible that some models may experience unexpected performance degradation. If this occurs, these optimizations can be disabled using the --disable-dge compiler command line option or the framework-level option ``additional_compile_opt=" --disable-dge"`` ---- Neuron Compiler [2.14.213.0] ---------------------------- Date: 07/03/2024 * Minor bug fixes and performance enhancements. * Improved flash attention kernel performance. ---- Neuron Compiler [2.13.72.0] ---------------------------- Date: 04/25/2024 * Minor bug fixes and enhancements. ---- Neuron Compiler [2.13.68.0] ---------------------------- Date: 04/10/2024 * This release fixes hang issues related to Triton Inference Server. ---- Neuron Compiler [2.13.66.0] ---------------------------- Date: 04/01/2024 * This release introduces a new ``--enable-mixed-precision-accumulation`` compiler option. This option instructs the compiler to perform intermediate calculations of reduction operators (such as the dot or reduce operators) in FP32 regardless of the operation's defined datatype. The final result of the operator will be cast from FP32 to the model-designated datatype (e.g., BF16). This helps to improve the operator's resulting accuracy. ---- Neuron Compiler [2.12.68.0] ---------------------------- Date: 01/18/2024 * Patch release with bug fixes. ---- Neuron Compiler [2.12.54.0] --------------------------- Date: 12/21/2023 * The compiler now generates instructions to check if a model references an embedding table with an illegal index. The check is made at model execution time. If an attempted invalid table index is encountered, the model execution will continue and the user will see an error similar to: WARNING: Received notification generated at runtime: failed to run scatter/gather (indirect memory copy with branch_label_id = xx), due to out-of-bound access. When this occurs, users are encouraged to review the model's gather/scatter input values to determine if there is a coding error. ---- Neuron Compiler [2.11.0.35] --------------------------- Date: 11/17/2023 * This release addresses performance related issues when training through ``neuronx-nemo-megatron`` library. ---- Neuron Compiler [2.11.0.34] ----------------------------- Date: 10/26/2023 * This release introduces the option-argument ``llm-training`` to the existing ``--distribution_strategy`` compiler option. This option-argument allows the compiler to make specific optimizations related to training distributed models. This new option-argument is equivalent to the previously introduced ``nemo`` option-argument, which will be deprecated in a future release. ---- Neuron Compiler [2.10.0.35] ----------------------------- Date: 09/26/2023 * This release addresses a compilation regression for certain configurations of Llama and Llama-2 inference models when it fails compilation with this error "IndirectLoad/Save requires contiguous indirect access per partition" . There is still a known issue for some configurations of the model with the error "Too many instructions after unroll for function sg0000" . To mitigate this, recompile using the ``--optlevel 1 (-O1)`` option. A complete fix will be coming in the future release which will not require this option ---- Neuron Compiler [2.10.0.34] ----------------------------- Date: 09/15/2023 * This release introduces a new ``--optlevel (-O)`` compiler option. This option allows the user to balance between compile-time and optimizations performed. Three levels are supported. Level ``--optlevel 1 (-O1)`` aims to minimize compile-time and allow for a more rapid model development cycle. Model execution time may be reduced. Level ``--optlevel 3 (-O3)`` performs whole-model optimization. This level will deliver the best performance however there will be longer compile-times and the compiler will use more host DRAM, potentially requiring a larger instance to compile the model. The default is ``--optlevel 2 (-O2)`` which provides a balance between model performance and compile time. The previous ``—enable-experimental-O1`` flag introduced in the 02/08/2023 Neuron Compiler [2.4.0.21] release is now deprecated. Using this flag will generate a message similar to: WARNING: Option —enable-experimental-O1 is deprecated and will be removed in a future release." Use ``--optlevel 1 (-O1)`` instead. ---- Neuron Compiler [2.9.0.16] ----------------------------- Date: 08/28/2023 * This release fixes an issue where any initial seed passed into the Random Number Generator operator was not honored. The RngBitGenerator operator now correctly accepts and uses setting the seed. Note that the current RNG implementation only supports 32-bit seeds. ---- Neuron Compiler [2.8.0.25] ----------------------------- Date: 07/19/2023 * This release introduces a new optional ``--distribution_strategy`` compiler option. This option informs the compiler what type of distributed APIs are used to shard the model and allows the compiler to make API-specific optimizations. Currently following option-arguments are supported: ``nemo``. ---- Neuron Compiler [2.7.0.40] ----------------------------- Date: 06/14/2023 * This release introduces a new ``--enable-saturate-infinity`` compiler option. A computation that can generate +/- infinity is at a high risk of generating Not-a-Number (NaN) values when the infinity value is used in subsequent computations. This option helps avoid this by converting +Inf/-Inf values to MAX/MIN_FLOAT before operations that could produce NaN values for +Inf/-Inf inputs on the target architecture. While this option helps to avoid NaN values, there is a potential performance degradation that occurs during model execution when this conversion is enabled. ---- Neuron Compiler [2.6.0.19] ----------------------------- Date: 05/01/2023 * This release introduces a new ``model-type`` option argument: ``unet-inference``. This option instructs the compiler to perform model-specific optimizations that produce executable models with improved performance on the specified target instance. * Added support for the HLO operator ``BitcastConvertType`` and also added support for ``TopK`` (sampling mode) operator. ---- Neuron Compiler [2.5.0.28] ----------------------------- Date: 03/28/2023 * This release introduces the ``trn1n`` option argument to the compiler ``target`` option to specify that it should generate code for a trn1n instance type. Example usage: ``neuronx-cc compile --target=trn1n ...`` * The compiler's usage message now includes the ``inf2`` option argument. * A new 8-bit floating point data type, ``fp8_e4m3``, is now supported and can be specified using the ``auto-cast-type`` option. This instructs the compiler to convert the FP32 operations selected via the ``--auto-cast`` option to a signed FP8 size with 4-bit exponent and 3-bit mantissa. Care must be taken to ensure that the down-casted values are representable within the 8-bit data range. ---- Neuron Compiler [2.4.0.21] ----------------------------- Date: 02/24/2023 * This release introduces the ``inf2`` option argument to the compiler ``target`` option to specify that it should generate code for an inf2 instance type. Example usage: ``neuronx-cc compile --target=inf2 ...`` The ``inf2`` option argument does not appear in the compiler's usage message. It will be added in the next release. ---- Neuron Compiler [2.4.0.21] ----------------------------- Date: 02/08/2023 * Added support for the following HLO operators: ``SelectAndScatter``. * Beta: ``--enable-experimental-O1`` flag: This option reduces the compile-time with a negligible impact on model execution performance. It allows the compiler to execute compiler passes in parallel to perform the compilation. By default the compiler uses 8 processes. This can be changed via the CLI option ``--num-parallel-jobs``. This option is expected to become the default in a future SDK release. ---- Neuron Compiler [2.3.0.4] ----------------------------- Date: 12/09/2022 * Added support for the following HLO operators: ``rev (reverse)``. * The ``pow()`` function can now handle both integer and floating-point exponents. * Optimization enhancements and bug fixes to improve model execution performance. ---- Neuron Compiler [2.2.0.73] ----------------------------- Date: 10/27/2022 * Adding support for the following HLO operators: ``LogicalNot``, ``atan2`` and ``DynamicUpdateSlice`` (for constant index). ---- Neuron Compiler [2.1.0.76] ----------------------------- Date: 10/5/2022 The Neuron Compiler is an Ahead-of-Time compiler that accelerates models for execution on NeuronCores. This release supports compiling models for training on a Trn1 instance using Pytorch Neuron. Users typically access the compiler via the Framework to perform model compilation, although it can also be run as a command line tool (*neuronx-cc*). The Neuron Compiler supports compiling models for mixed precision calculations. The trn1 hardware supports matrix multiplication using FP16, BF16, and FP32 on its Matrix Multiplication Engine, and accumulations using FP32. Operators such as activations or vector operations are supported using FP16, BF16, and FP32. Tensor transpose can be accomplished in FP16, BF16, FP32, or TF32 datatypes. By default, scalar and vector operations on FP32 values will be done in FP32, while matrix multiplications are cast to BF16 and transpose operations are cast to FP32. This default casting will generate the highest performance for a FP32 trained model. By default, the compiler will target maximum performance by automatically casting the model to mixed precision. It also provides an option (``--auto-cast``) that allows the user to make tradeoffs between higher performance and optimal accuracy. The decision on what option argument to use with the ``--auto-cast`` option will be application specific. Compiler CLI options can be passed to the compiler via the framework. Known issues ~~~~~~~~~~~~ - The Random Number Generator operation can be passed an initial seed value, however setting the seed is not supported in this release. - The exponent value of the pow() function must be a compile-time integer constant. - The compiler treats INT64 datatypes as INT32 by truncating the high-order bits. If possible, cast these values to 32 bits . - Model compilation time is proportional to the model size and operators used. For some larger NLP models it may be upwards of 30 minutes. ---- Supported Operators ------------------- The following XLA operators are supported by the Neuron Compiler. Future releases will broaden model support by providing additional XLA operators defined in https://www.tensorflow.org/xla/operation_semantics. The list of supported operators can also be retrieved from the command line using :ref:`neuronx-cc list-operators`. +-------------------------+-------------------------------------------+ | Supported XLA Operators | Notes | +=========================+===========================================+ | Abs | | +-------------------------+-------------------------------------------+ | Add | | +-------------------------+-------------------------------------------+ | Allgather | | +-------------------------+-------------------------------------------+ | Allreduce | | +-------------------------+-------------------------------------------+ | Atan2 | | +-------------------------+-------------------------------------------+ | Batchnorm | | +-------------------------+-------------------------------------------+ | Batchnormgrad | | +-------------------------+-------------------------------------------+ | Batchnorminference | | +-------------------------+-------------------------------------------+ | BitcastConvertType | | +-------------------------+-------------------------------------------+ | Broadcast | | +-------------------------+-------------------------------------------+ | BroadcastInDim | | +-------------------------+-------------------------------------------+ | Ceil | | +-------------------------+-------------------------------------------+ | Clamp | | +-------------------------+-------------------------------------------+ | Compare | | +-------------------------+-------------------------------------------+ | Concatenate | | +-------------------------+-------------------------------------------+ | Constant | | +-------------------------+-------------------------------------------+ | ConstantLiteral | | +-------------------------+-------------------------------------------+ | ConvertElementType | | +-------------------------+-------------------------------------------+ | Cos | | +-------------------------+-------------------------------------------+ | Customcall | | +-------------------------+-------------------------------------------+ | Div | | +-------------------------+-------------------------------------------+ | Dot | | +-------------------------+-------------------------------------------+ | DotGeneral | | +-------------------------+-------------------------------------------+ | DynamicUpdateSlice | Supports only for constant index | +-------------------------+-------------------------------------------+ | Eq | | +-------------------------+-------------------------------------------+ | Exp | | +-------------------------+-------------------------------------------+ | Floor | | +-------------------------+-------------------------------------------+ | Gather | Supports only disjoint start_index_map | | | and remapped_offset_dims | +-------------------------+-------------------------------------------+ | Ge | | +-------------------------+-------------------------------------------+ | GetTupleElement | | +-------------------------+-------------------------------------------+ | Gt | | +-------------------------+-------------------------------------------+ | Iota | | +-------------------------+-------------------------------------------+ | Le | | +-------------------------+-------------------------------------------+ | Log | | +-------------------------+-------------------------------------------+ | LogicalAnd | | +-------------------------+-------------------------------------------+ | LogicalNot | | +-------------------------+-------------------------------------------+ | Lt | | +-------------------------+-------------------------------------------+ | Max | | +-------------------------+-------------------------------------------+ | Min | | +-------------------------+-------------------------------------------+ | Mul | | +-------------------------+-------------------------------------------+ | Ne | | +-------------------------+-------------------------------------------+ | Neg | | +-------------------------+-------------------------------------------+ | Pad | | +-------------------------+-------------------------------------------+ | Pow | Exponent argument must be a compile-time | | | integer constant | +-------------------------+-------------------------------------------+ | Reduce | Min, Max, Add and Mul are the only | | | supported computations. Init_values must | | | be constant | +-------------------------+-------------------------------------------+ | Reshape | | +-------------------------+-------------------------------------------+ | Rev (reverse) | | +-------------------------+-------------------------------------------+ | RngBitGenerator | Ignores user seed | +-------------------------+-------------------------------------------+ | RngUniform | | +-------------------------+-------------------------------------------+ | Rsqrt | | +-------------------------+-------------------------------------------+ | Scatter | | +-------------------------+-------------------------------------------+ | Select | | +-------------------------+-------------------------------------------+ | SelectAndScatter | | +-------------------------+-------------------------------------------+ | ShiftRightLogical | | +-------------------------+-------------------------------------------+ | Sign | | +-------------------------+-------------------------------------------+ | Sin | | +-------------------------+-------------------------------------------+ | Slice | | +-------------------------+-------------------------------------------+ | Sqrt | | +-------------------------+-------------------------------------------+ | Sub | | +-------------------------+-------------------------------------------+ | Tanh | | +-------------------------+-------------------------------------------+ | Transpose | | +-------------------------+-------------------------------------------+ | Tuple | | +-------------------------+-------------------------------------------+