Skip to main content
Ctrl+K
Neuron 2.32.0 is released! Check the What's New and Release Notes for more details.
Logo image
Ctrl+K
Search Engine: Default Google
  • About Neuron
    • Ask Amazon AI Helper Tools
    • Benchmarks
      • Inf1 Inference Performance
      • Inf2 Inference Performance
      • Trn1/Trn1n Inference Performance
      • Trn1/Trn1n Training Performance
    • Calculator
    • Beta Participation
    • Generative LLM Inference on Neuron
    • Model Samples
      • Training on Trn1
      • Inference on Inf2/Trn1/Trn2
      • Inference on Inf1
    • Neuron FAQ
    • Neuron Features
      • Custom C++ operators
      • Data types
      • Logical NeuronCore configuration
      • Neuron persistent cache
      • NeuronCore batching
      • NeuronCore pipeline
      • Rounding modes
    • Open Source
    • SDK Maintenance Policy
    • Security
    • Term Glossary
    • Troubleshooting
    • What is AWS Neuron?
    • Older Neuron FAQS
      • ONNX FAQ
      • Contributing Guidelines FAQ
      • Inference with Neuron FAQ
      • Troubleshooting for Inf1 FAQ
      • Training with Neuron FAQ
  • Neuron Architecture
    • AWS Inferentia
    • AWS Inferentia2
    • AWS Trainium
    • AWS Trainium2
    • AWS Trainium3
    • NeuronCore v1
    • NeuronCore v2
    • NeuronCore v3
    • NeuronCore v4
    • Inf1 Architecture
    • Inf2 Architecture
    • Trn1 Architecture
    • Trn2 Architecture
    • Trn3 Architecture
  • What's New
    • Release Notes
      • Neuron 2.32.0
        • NKI
        • NKI Library
        • Neuron Compiler
        • Neuron Runtime
        • Developer tools
        • Neuron Agentic Development
        • JAX NeuronX
        • vLLM Neuron
        • NxD Inference
        • Deep Learning AMIs
        • Deep Learning Containers
      • Neuron 2.31.1
      • Neuron 2.31.0
        • NKI
        • NKI Library
        • Neuron Compiler
        • Neuron Runtime
        • Developer tools
        • Neuron Agentic Development
        • PyTorch NeuronX
        • JAX NeuronX
        • NxD Core
        • NxD Inference
        • NxD Training
        • Deep Learning AMIs
        • Deep Learning Containers
      • Neuron 2.30.0
        • NKI
        • NKI Library
        • Neuron Compiler
        • Neuron Runtime
        • Developer tools
        • Neuron Agentic Development
        • Deep Learning AMIs
        • Deep Learning Containers
        • JAX
      • Previous Neuron versions
        • Neuron 2.31.0
        • Neuron 2.30.0
        • Neuron 2.29.1
        • Neuron 2.29.0
        • Neuron 2.28.1
        • Neuron 2.28.0
        • Neuron 2.27.1
        • Neuron 2.27.0
        • Neuron 2.26.1
        • Neuron 2.26.0
        • Neuron 2.25.0
      • Component release notes
        • Neuron Compiler
        • Neuron Agentic Development
        • Neuron Containers
        • Neuron Developer Tools
        • Neuron DLAMI
        • JAX NeuronX
        • NKI Library
        • Neuron Kernel Interface
        • NxD Core
        • NxD Inference
        • NxD Training
        • vLLM Neuron
        • PyTorch Neuron Framework
        • Neuron Runtime
        • Older components and features
      • Release artifacts
  • Announcements
  • News & Blogs
  • Contribute

Get Started

  • Quickstarts
    • Quickstart: Train a Model on Trainium
    • Quickstart: Run Inference on Inferentia
    • Quickstart: Serve models online with vLLM on Neuron
    • Quickstart: Run offline inference with vLLM on Neuron
    • Ask Amazon AI helper tools
    • Neuron GitHub Samples
  • Set Up Environments
    • PyTorch
      • New DLAMI
      • Update Existing DLAMI
      • New DLC
      • Update Existing DLC
      • New Manual Configuration
      • Update Manual Configuration
    • JAX
      • Install JAX via Deep Learning AMI
      • Install JAX via Deep Learning Container
      • Install JAX Manually
    • Multi-framework
    • Troubleshooting
  • Deploy Model Workloads
    • Deployment Overview
    • Pre-configured Environments
      • Deep Learning AMIs
      • DLC Images
      • Customize DLC
      • Docker Setup
      • Quickstart: Deploy DLC with vLLM
    • Use Amazon EC2
      • Train on EC2
      • Inference (Inf2, Trn1)
      • Inference with DLC
      • Inference (Inf1)
    • Use Amazon EKS
      • Set up EKS for Neuron
      • Components Included
      • Inference on EKS
      • Training on EKS
      • Dynamic Resource Allocation
        • Support Files
      • Neuron UltraServer Operator
      • UltraServer MPI Jobs
      • EKS Prerequisites
    • Use Amazon ECS
      • Inference on ECS
      • Training on ECS
      • Node Problem Detector
    • Use AWS Batch
      • Train on AWS Batch
    • Use AWS ParallelCluster
      • Train on ParallelCluster
    • Use Amazon SageMaker
      • Train on SageMaker
    • Infrastructure Components
      • Plugins Overview
      • Scheduler Extension
      • Scheduler Flow
      • Neuron Monitor
      • Node Problem Detector
      • NPD Permissions (IRSA)
    • Tutorials
      • Docker Environment Setup
      • Build and Run Containers
      • OCI Hook Setup
      • Inference Tutorials
        • Run Inference in PyTorch Neuron Container
        • Deploy a TensorFlow Resnet50 model as a Kubernetes service
      • Training Tutorials
        • Run Training in PyTorch Neuron Container
        • Deploy a simple mlp training script as a Kubernetes job
    • Docker Examples
      • Inference DLC Dockerfile
      • Libmode Dockerfile
      • TF Serving Dockerfile
      • TorchServe Config
      • Libmode Entrypoint
      • TorchServe Neuron
      • Training DLC Dockerfile
      • MLP Training Example
      • v1 App+RT Different Containers
      • v1 App+RT Same Container
      • v1 Neuron Runtime
      • v1 Torch Neuron
      • v1 Entrypoint
    • Third-party Solutions
    • FAQ
    • Troubleshooting
    • Container Release Notes
    • DLAMI Release Notes

ML Frameworks

  • ML Framework Support
  • PyTorch
    • About PyTorch on Neuron
    • Native PyTorch
    • PyTorch Setup
    • Inference
      • Tutorials
        • Compiling and Deploying HuggingFace Pretrained BERT on Trn1 or Inf2
        • BERT TorchServe Tutorial
        • LibTorch C++ Tutorial
        • Compiling and Deploying ResNet50 on Trn1 or Inf2
        • T5 model inference on Trn1 or Inf2
      • Additional Examples
        • AWS Neuron Samples GitHub Repository
        • Transformers Neuron GitHub samples
      • API Reference Guide
        • PyTorch NeuronX Tracing API for Inference
        • PyTorch Neuron (torch-neuronx) Weight Replacement API for Inference
        • PyTorch NeuronX NeuronCore Placement APIs
        • PyTorch NeuronX Analyze API for Inference
        • PyTorch NeuronX DataParallel API
      • Developer Guide
        • NeuronCore Allocation and Model Placement for Inference (torch-neuronx)
        • Comparison of Traced Inference versus XLA Lazy Tensor Inference (torch-neuronx)
      • Misc
        • Component Release Notes for Neuron PyTorch Framework
    • torch-neuron v. torch-neuronx
    • Release Notes
    • Supported operators
    • Profiling API
    • Profiling developer guide
    • neuron_parallel_compile CLI
    • Multi-node EFA setup
  • JAX
    • JAX NeuronX plugin Setup
    • JAX NeuronX Known Issues
    • API Reference Guide for JAX Neuronx
      • JAX NeuronX Environment Variables
    • Release Notes

Training

  • NxD Core (Training)
    • Setup
    • App Notes
      • Tensor Parallelism Overview
      • Pipeline Parallelism Overview
      • Activation Memory Reduction
      • Context Parallelism Overview
    • API Reference Guide
      • Distributed Strategies APIs
      • Training APIs
      • Inference APIs
      • ModelBuilderV2 API Reference
    • Developer Guide
      • Developer guide for Tensor Parallelism
      • Developer guide for Pipeline Parallelism
      • Developer guide for Activation Memory reduction
      • Developer guide for save/load checkpoint
      • Developer guide for Neuron-PT-Lightning
      • Developer guide for model and optimizer wrapper
      • Developer guide for LoRA finetuning
    • Tutorials
      • Training Tutorials
        • Training using Tensor Parallelism
        • Training Llama 3.1 8B/Llama 3 8B using TP and ZeRO-1
        • Training Llama 3.1 70B/Llama 3 70B using TP and PP
        • Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning
      • Inference Tutorials
        • [Broken] T5 inference with Tensor Parallelism
    • Misc
      • Component Release Notes for NxD Core

Inference

  • Inference on Neuron
  • NxD Inference + vLLM
    • Overview
    • Setup
    • vLLM
      • Quickstart: Offline Model Serving
      • Quickstart: Online Model Serving
      • vLLM on Neuron User Guide
      • Model Recipes
        • Llama 3.3 70B
        • Qwen3 235B A22B
      • Deploy Llama4 with vLLM
    • Tutorials
      • Disaggregated Inference (1P1D)
      • Disaggregated Inference
      • Flux Inference
      • Flux Inpainting
      • Benchmark using Performance CLI
      • GPT-OSS 120B
      • Llama3.1 405B on Trn2
      • Llama3.1 405B with Speculative Decoding
      • Llama3.1 70B Instruct Accuracy Evaluation
      • Llama3.1 8B with Multi-LoRA
      • Llama3.3 70B FP8
      • Llama3.3 70B with APC
      • Llama3.3 70B with Data Parallelism
      • Llama3.3 70B with Speculative Decoding
      • Llama4
      • Llama4 Legacy
      • Pixtral
      • Qwen3 MoE 235B
      • Qwen2 VL 7B
      • Speculative Decoding
      • Qwen3-VL 8B
    • Developer Guides
      • Accuracy Evaluation
      • Custom Quantization
      • Disaggregated Inference
      • Feature Guide
      • Using Pipeline Execution Mode
      • LLM Benchmarking
      • Migrate from TNX
      • Model Reference
      • MoE Architecture
      • Migrate from NxD Core
      • Onboarding Models
      • Performance Benchmarking CLI
      • vLLM Guide (Legacy)
      • vLLM Guide v1
      • Weights Sharding
      • Writing Tests
    • API Reference Guide
      • NxD Inference API Reference
    • App Notes
      • Parallelism Techniques for LLM Inference
    • Release Notes
    • Misc
      • Component Release Notes for NxD Inference
      • Troubleshooting Guide for NxD Inference
  • NxD Core (Inference)
    • Setup
    • App Notes
      • Tensor Parallelism Overview
      • Pipeline Parallelism Overview
      • Activation Memory Reduction
      • Context Parallelism Overview
    • API Reference Guide
      • Distributed Strategies APIs
      • Training APIs
      • Inference APIs
      • ModelBuilderV2 API Reference
    • Developer Guide
      • About NeuronX-Distributed (NxD) Inference
    • LoRA Guide
    • Tutorials
      • Training Tutorials
        • Training using Tensor Parallelism
        • Training Llama 3.1 8B/Llama 3 8B using TP and ZeRO-1
        • Training Llama 3.1 70B/Llama 3 70B using TP and PP
        • Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning
      • Inference Tutorials
        • [Broken] T5 inference with Tensor Parallelism
    • Misc
      • Component Release Notes for NxD Core
  • vLLM Neuron (Beta)
    • Get started
      • Setting up
      • Online serving quickstart
      • Offline serving quickstart
      • Migrating from NxDI
    • Deploy & serve
      • Features guide
      • Configuration reference
      • Profiling workloads
      • Feature–model compatibility
    • Model Recipes
      • Llama 3
      • GPT-OSS
      • Qwen3-VL
      • Qwen3-Embedding
    • Tutorials
      • Deploying Llama 3.3 70B
      • EAGLE3 speculative decoding (Llama 3.1)
      • EAGLE3 speculative decoding (GPT-OSS)
      • Disaggregated inference (1P1D and xPyD)
      • Deploying gpt-oss
      • Benchmarking prefix caching (GPT-OSS)
      • Deploying Qwen3-VL-32B
      • Disaggregated encoder (1E1PD and xEyPD)
      • Deploying Qwen3-Embedding-8B
    • Model development
      • Onboarding a model
      • Onboarding a vision-language model
      • Optimizing a vision-language model
      • CPU development workflow
      • NKI CPU simulator
      • Debugging model code
      • Debugging accuracy issues
      • Accuracy debugger tools
    • Concepts & architecture
      • Parallelism
        • Attention DP
        • Component-Level DP Sharding
        • Data Parallelism (DP)
        • DCP (Decode Context Parallelism)
        • Expert Parallelism (EP)
        • Tensor Parallelism (TP)
        • Vision Encoder Parallelism
      • Speculation
        • Speculative Decoding (EAGLE3) in the vLLM Neuron Framework
      • Multimodal
        • Block Packing Vision Attention
        • On-Device Encoder Cache
        • M-RoPE (Multimodal Rotary Position Embedding)
      • vLLM integration
        • vLLM Integration Design Reference
        • KV Caching with vLLM
        • Additional Configuration Options
        • Async Scheduling and Async Execution
        • Decode Context-Length Bucketing
        • determine_available_memory design
        • KV Cache Quantization
        • vLLM Metrics Integration
        • Runtime NRT Profiling via vLLM’s Profiler Interface
        • Scheduler Design
        • Padding and Batching with Block KV
        • Prefill Processing
        • Pooling models on Trainium
        • Prompt embeddings on Neuron
        • Structured Outputs and Tool Calling
        • Disaggregated Inference — Design Document
      • Framework
        • Asynchronous Execution Framework Capability
        • Checkpoint Loading Framework Capability
        • Canonical Model — Design & Porting Guide
        • Scalable Model Registry Design
      • Compilation
        • Compilation Cache
        • CPU Compilation
        • FX Passes Architecture Design
        • Aliasing Output Rewrite Pass
        • FX Device Rewriting for XLA Compilation
        • In-Place to Out-of-Place Rewrite Pass
      • Accuracy
        • Accuracy Debugging Design
        • Dataset Evaluation Design
        • KV Cache Analysis Design
        • vLLM Neuron Logit Validation Design
        • Module-Level Accuracy Test Guidelines
        • Tensor Capture Design
        • Tensor Compare Design
        • Tensor Replacement — Design
        • Input Snapshot Design

Developer Tools

  • Neuron Developer Tools
    • System Tools
      • Neuron-Monitor User Guide
      • Neuron-Top User Guide
      • Neuron-LS User Guide
      • Neuron-Sysfs User Guide
      • NCCOM-TEST User Guide
    • Third-party Tools
    • Tutorials
      • Profiling a vLLM Inference Workload on AWS Trainium
      • Track System Resource Utilization during Training with neuron-monitor using PyTorch Neuron
    • Release Notes
  • Neuron Agentic Development
    • Getting Started
    • Developer Guides
      • Autoport NxDI Deep Dive
      • Autoport vLLM Neuron Deep Dive
      • Equivalence Validation Deep Dive
    • Tutorials
      • Port a model with Autoport
      • Port a model to vLLM Neuron
      • Validate a port with Equivalence
  • Neuron Explorer
    • Get Started
    • Capture Profiles
    • UI Overview
    • Device Trace Viewer
    • System Trace Viewer
    • Hierarchy Viewer
    • Source Code Viewer
    • Summary Viewer
    • Tensor Viewer
    • Memory Viewer
    • Region Highlighter
    • AI Recommendation Viewer
    • Profile Parquet Schema Reference
    • Analyze Profile Output
    • View Profiles with Perfetto
    • Neuron Profiler to Neuron Explorer Migration Guide
    • Troubleshooting & FAQs
    • Environment Variables
    • Glossary

Compiler, Runtime, & Collectives

  • Graph Compiler
    • About neuronx-cc
    • NeuronX Compiler for Trn* & Inf2
      • API Reference Guide
      • How-to: Generate a NEFF file
      • How-to: Convolution
        • * `UNet training sample <https://github.com/aws-neuron/aws-neuron-samples/tree/master/torch-neuronx/training/unet_image_segmentation>`_ - Sample UNet training implementation
      • FAQ
    • Error Codes
      • NCC_EARG001
      • NCC_EBIR023
      • NCC_EBVF030
      • NCC_EHCA005
      • NCC_EMOD025
      • NCC_EOOM001
      • NCC_EOOM002
      • NCC_ESFH002
      • NCC_ESPP004
      • NCC_ESPP047
      • NCC_EUOC002
      • NCC_EVRF001
      • NCC_EVRF004
      • NCC_EVRF005
      • NCC_EVRF006
      • NCC_EVRF007
      • NCC_EVRF009
      • NCC_EVRF010
      • NCC_EVRF011
      • NCC_EVRF013
      • NCC_EVRF015
      • NCC_EVRF016
      • NCC_EVRF017
      • NCC_EVRF018
      • NCC_EVRF019
      • NCC_EVRF022
      • NCC_EVRF031
      • NCC_EVRF035
      • NCC_EVRF036
      • NCC_EVRF037
      • NCC_EVRF038
      • NCC_EVRF039
      • NCC_EVRF040
      • NCC_EVRF041
      • NCC_EVRF042
      • NCC_EVRF043
      • NCC_EVRF044
      • NCC_EVRF045
      • NCC_EVRF046
      • NCC_EVRF047
      • NCC_EVRF048
      • NCC_EVRF049
      • NCC_EVRF050
      • NCC_EVRF051
      • NCC_EVRF052
      • NCC_EVRF053
      • NCC_EVRF054
      • NCC_EVRF055
      • NCC_EVRF056
      • NCC_EVRF057
      • NCC_EVRF058
      • NCC_EVRF059
      • NCC_EVRF060
      • NCC_EVRF061
      • NCC_EVRF062
      • NCC_EVRF063
      • NCC_EVRF064
      • NCC_EVRF065
      • NCC_EXSP001
      • NCC_EXTP004
    • Release Notes
    • neuron-cc (Legacy)
      • API Reference Guide
        • Neuron compiler CLI Reference Guide (neuron-cc)
      • CLI Reference
      • Developer Guide
      • FAQ
  • Neuron Runtime
    • Overview
    • Get Started
    • Deep Dives
      • Compute-Communication Overlap
      • Neuron Device Memory
      • Direct HBM Tensor Allocation
      • Runtime Performance Tips
      • Neuron Runtime Core Dumps
      • Inter-node Collectives
      • Intra-node Collectives
    • How-To Guides
      • Developer guide
      • Migrate to the explicit async APIs
      • Configuration guide
    • API Reference
      • NRT API
      • NRT Status
      • NRT Version
      • Debug Stream APIs
      • NRT Async APIs Overview
      • NRT Async APIs Best Practices
      • NRT Async APIs Examples
      • NRT Async
      • NRT Async Send/Recv
      • NRT Profile
      • NRT System Trace
      • Debug Stream
      • NEC API
      • NDL API
      • Neuron Driver Shared
      • Tensor Batch Operations
      • Neuron Datastore
      • NRT Experimental
    • nrtpy (Neuron Runtime Python)
      • Getting started with nrtpy
      • Test and debug a NKI kernel end-to-end
      • Validate, benchmark, and trace a kernel
      • Troubleshooting
      • Model API
      • Tensor API
      • Configuration
      • Error handling
    • NRT Debug Stream
    • Troubleshooting on Inf1 and Trn1
    • Release Notes
    • FAQ
  • Collectives
  • Neuron C++ Custom Operators
    • API Reference Guide
      • Custom Operators API Reference Guide [Beta]
    • Developer Guide
      • Neuron Custom C++ Operators Developer Guide [Beta]
    • Tutorials
      • Neuron Custom C++ Operators in MLP Training
      • Neuron Custom C++ Operators Performance Optimization
    • Misc (Neuron Custom C++ Operators)
      • Neuron Custom C++ Tools Release Notes
      • Neuron Custom C++ Library Release Notes

Neuron Kernel Interface (NKI)

  • NKI
    • Get started
      • Environment Setup
      • First NKI Kernel
      • NKI Language Guide
      • Concepts
        • Memory Hierarchy
        • Data Representation
        • Indexing
        • Tiling
        • Direct Memory Access
        • Logical Neuron Cores
    • Guides
      • Tutorials
        • Matrix Multiplication
        • AveragePool2D
        • Transpose2D
        • Fused Mamba
        • Introduction to NKI Kernel Optimization
      • Architecture
        • Trainium/Inferentia2 Guide
        • Trainium2 Guide
        • Trainium3 Guide
      • NKI CPU Simulator
      • Insert NKI Kernels into Models
      • Use NKI Scheduling APIs
      • Profile a NKI Kernel
    • Deep dives
      • Performance Optimizations
      • MXFP8/4 Matrix Multiplication
      • NKI Access Patterns
      • NKI Dynamic Loops
      • Descriptor Generation Engine (DGE)
      • DMA Bandwidth Guide
      • About the NKI Compiler
    • Migration
    • NKI FAQ
  • NKI API Reference
    • nki
      • nki.jit
      • nki.simulate
    • nki.isa
      • nki.isa.nc_matmul
      • nki.isa.nc_matmul_mx
      • nki.isa.nc_transpose
      • nki.isa.activation
      • nki.isa.activation_reduce
      • nki.isa.activate2
      • nki.isa.tensor_reduce
      • nki.isa.tensor_partition_reduce
      • nki.isa.tensor_tensor
      • nki.isa.tensor_tensor_scan
      • nki.isa.scalar_tensor_tensor
      • nki.isa.tensor_scalar
      • nki.isa.tensor_scalar_reduce
      • nki.isa.tensor_scalar_cumulative
      • nki.isa.tensor_copy
      • nki.isa.tensor_copy_predicated
      • nki.isa.exponential
      • nki.isa.reciprocal
      • nki.isa.quantize_mx
      • nki.isa.iota
      • nki.isa.dropout
      • nki.isa.affine_select
      • nki.isa.range_select
      • nki.isa.select_reduce
      • nki.isa.sequence_bounds
      • nki.isa.memset
      • nki.isa.bn_stats
      • nki.isa.bn_aggr
      • nki.isa.local_gather
      • nki.isa.nc_n_gather
      • nki.isa.dma_copy
      • nki.isa.dma_transpose
      • nki.isa.dma_compute
      • nki.isa.max8
      • nki.isa.nc_find_index8
      • nki.isa.nc_match_replace8
      • nki.isa.nc_stream_shuffle
      • nki.isa.register_alloc
      • nki.isa.register_load
      • nki.isa.register_move
      • nki.isa.register_store
      • nki.isa.core_barrier
      • nki.isa.sendrecv
      • nki.isa.rng
      • nki.isa.rand2
      • nki.isa.rand_set_state
      • nki.isa.rand_get_state
      • nki.isa.set_rng_seed
      • nki.isa.nonzero_with_count
      • nki.isa.topk
      • nki.isa.engine
      • nki.isa.dma_engine
      • nki.isa.reduce_cmd
      • nki.isa.dge_mode
      • nki.isa.oob_mode
      • nki.isa.matmul_perf_mode
      • nki.isa.nc_version
      • nki.isa.get_nc_version
      • nki.isa.VirtualRegister
      • nki.isa.NkiInstruction
    • nki.language
      • nki.language.ndarray
      • nki.language.zeros
      • nki.language.ones
      • nki.language.full
      • nki.language.zeros_like
      • nki.language.empty_like
      • nki.language.shared_constant
      • nki.language.shared_identity_matrix
      • nki.language.rand
      • nki.language.random_seed
      • nki.language.NkiTensor
      • nki.language.load
      • nki.language.load_transpose2d
      • nki.language.store
      • nki.language.copy
      • nki.language.matmul
      • nki.language.transpose
      • nki.language.abs
      • nki.language.abs_max
      • nki.language.abs_min
      • nki.language.add
      • nki.language.arctan
      • nki.language.ceil
      • nki.language.cos
      • nki.language.exp
      • nki.language.floor
      • nki.language.log
      • nki.language.maximum
      • nki.language.minimum
      • nki.language.multiply
      • nki.language.negative
      • nki.language.power
      • nki.language.reciprocal
      • nki.language.rsqrt
      • nki.language.sign
      • nki.language.sin
      • nki.language.sqrt
      • nki.language.square
      • nki.language.subtract
      • nki.language.tan
      • nki.language.tanh
      • nki.language.trunc
      • nki.language.relu
      • nki.language.sigmoid
      • nki.language.silu
      • nki.language.silu_dx
      • nki.language.gelu
      • nki.language.gelu_dx
      • nki.language.gelu_apprx_sigmoid
      • nki.language.gelu_apprx_sigmoid_dx
      • nki.language.gelu_apprx_tanh
      • nki.language.mish
      • nki.language.softplus
      • nki.language.softmax
      • nki.language.erf
      • nki.language.erf_dx
      • nki.language.dropout
      • nki.language.rms_norm
      • nki.language.all
      • nki.language.max
      • nki.language.mean
      • nki.language.min
      • nki.language.prod
      • nki.language.sum
      • nki.language.var
      • nki.language.equal
      • nki.language.not_equal
      • nki.language.less
      • nki.language.less_equal
      • nki.language.greater
      • nki.language.greater_equal
      • nki.language.logical_and
      • nki.language.logical_or
      • nki.language.logical_xor
      • nki.language.logical_not
      • nki.language.bitwise_and
      • nki.language.bitwise_or
      • nki.language.bitwise_xor
      • nki.language.invert
      • nki.language.left_shift
      • nki.language.right_shift
      • nki.language.broadcast_to
      • nki.language.ds
      • nki.language.expand_dims
      • nki.language.where
      • nki.language.gather_flattened
      • nki.language.affine_range
      • nki.language.dynamic_range
      • nki.language.sequential_range
      • nki.language.static_range
      • nki.language.fori_loop
      • nki.language.while_loop
      • nki.language.psum
      • nki.language.sbuf
      • nki.language.hbm
      • nki.language.private_hbm
      • nki.language.shared_hbm
      • nki.language.is_psum
      • nki.language.is_sbuf
      • nki.language.is_hbm
      • nki.language.is_on_chip
      • nki.language.device_print
      • nki.language.no_reorder
      • nki.language.program_id
      • nki.language.num_programs
      • nki.language.program_ndim
      • nki.language.bool_
      • nki.language.int8
      • nki.language.int16
      • nki.language.int32
      • nki.language.uint8
      • nki.language.uint16
      • nki.language.uint32
      • nki.language.float16
      • nki.language.float32
      • nki.language.bfloat16
      • nki.language.tfloat32
      • nki.language.float8_e4m3
      • nki.language.float8_e5m2
      • nki.language.float8_e4m3fn
      • nki.language.float8_e8m0fnu
      • nki.language.float8_e5m2_x4
      • nki.language.float8_e4m3fn_x4
      • nki.language.float4_e2m1fn_x4
      • nki.language.tile_size
      • nki.language.prelu
      • nki.language.bypass
    • nki.collectives
      • nki.collectives.all_reduce
      • nki.collectives.all_gather
      • nki.collectives.all_gather_v
      • nki.collectives.reduce_scatter
      • nki.collectives.all_to_all
      • nki.collectives.all_to_all_v
      • nki.collectives.collective_permute
      • nki.collectives.collective_permute_implicit
      • nki.collectives.collective_permute_implicit_reduce
      • nki.collectives.collective_permute_implicit_current_processing_rank_id
      • nki.collectives.rank_id
      • nki.collectives.ReplicaGroup
    • NKI ISA API Common Fields
  • NKI Library
    • Get started
    • Overview
    • Kernel Design Specs
      • RMSNorm-Quant
    • Kernel API Reference
      • Argsort Unstable
      • Attention Block TKG
      • Attention CTE
      • Attention KV-Parallel Segmented CTE
      • Attention MXFP8 TKG
      • Attention Segmented CTE
      • Attention TKG
      • Blockwise MM Backward
      • Blockwise MM Backward MXFP8
      • Blockwise MM Forward MXFP8
      • Blockwise MM Shard-on-Block
      • Build All-to-All-V Metadata
      • Build All2All Dispatch Metadata
      • Collective Communication Kernels
      • Conv1D
      • Conv3D
      • Conv3D Temporal Unroll
      • Conv3D Transpose
      • Cross Entropy
      • Cumsum
      • Depthwise Conv1D
      • Dynamic Elementwise Add
      • FGCC
      • Find Nonzero Indices
      • Fine-Grained All-Gather
      • Foreach Elementwise
      • Foreach Norm
      • FP8 Quantize
      • Fused Adam
      • Gather
      • GPSIMD Top-K
      • Linear Scan
      • Matmul MXFP8
      • Matmul MXFP8 Backward
      • MLA QKV CTE
      • MLA Sparse Attention CTE
      • MLA V-Up O-Proj CTE
      • MLP
      • MLP Backward MXFP8
      • MLP Forward MXFP8
      • MoE CTE
      • MoE TKG
      • MS Deformable Attention
      • MS Deformable Attention Backward
      • MX MoE Block TKG Wrapper
      • Output Projection CTE
      • Output Projection TKG
      • Pad
      • Permute A2AV
      • Permute Routed Tokens
      • QKV
      • QKV Batch Shard
      • QKV CTE MLA
      • Quantize MXFP8
      • Ring Attention Backward
      • Ring Attention Forward
      • Ring Attention Unpermute
      • RMSNorm-Quant
      • RMSNorm MX Prefill
      • RMSNorm Router Top-K A2AV
      • RMSNorm Router Top-K TKG
      • RNG
      • RoPE
      • Router Top-K
      • SBUF-to-SBUF All-Gather
      • Scatter-Add
      • Selective Scan
      • Sparse Attention Indexer MX BF16Score
      • SSD
      • SWA Fused CTE
      • Top-K Reduce
      • Transformer TKG
      • Unpermute A2AV
    • Kernel Utilities
      • SbufManager (Allocator)
      • TensorView
    • Release Notes

Archive

  • Archived content
    • Fine-tune T5 model on Trn1
    • Running SSD300 with AWS Neuron
    • Megatron GPT Pretraining
    • Launch a GPT-3 pretraining job using neuronx-nemo-megatron
    • Training GPT-NeoX 20B with Tensor Parallelism and ZeRO-1 Optimizer
    • Fine-tuning Llama2 7B with tensor parallelism and ZeRO-1 optimizer using Neuron PyTorch-Lightning
    • Training Llama-2-7B/13B/70B using Tensor Parallelism and Pipeline Parallelism with Neuron PyTorch-Lightning
    • Training CodeGen2.5 7B with Tensor Parallelism and ZeRO-1 Optimizer
    • Profiling Multi-Node Training Jobs with Neuron Explorer
    • Training GPT-NeoX 6.9B with Tensor Parallelism and ZeRO-1 Optimizer
    • Neuron application notes
      • Neuron Runtime Library
      • Mixed precision (neuron-cc)
      • Mixed precision (neuronx-cc)
      • Introducing NxD Inference
      • Introducing NxD Training
      • Performance
      • Parallel execution
      • PyTorch for Neuron
        • Running inference on variable input shapes with bucketing
        • Running R-CNNs on Inf1
        • Data Parallel Inference on Torch Neuron
      • PyTorch for NeuronX
        • Introducing PyTorch 2.6 Support
        • Introducing PyTorch 2.7 Support
        • Introducing PyTorch 2.8 Support
        • Introducing PyTorch 2.9 Support
        • Introducing PyTorch 2.5 Support
        • Migration From XLA_USE_BF16/XLA_DOWNCAST_BF16
        • Data Parallel Inference on torch_neuronx
        • Graph Partitioner on torch_neuronx
    • Neuron Plugin for TensorBoard (Inf1)
    • TensorBoard plugin (archived)
      • NeuronX Plugin for TensorBoard (Trn1) (archived)
      • Neuron Plugin for TensorBoard (Inf1)
      • Profiling PyTorch NeuronX with TensorBoard (archived)
      • Track Training Progress in TensorBoard using PyTorch Neuron (archived)
    • NeuronPerf (Beta)
      • NeuronPerf Install
      • Overview
      • Terminology
      • Examples
      • Benchmark Guide
      • Evaluate Guide
      • Compile Guide
      • Model Index Guide
      • API
      • Framework Notes
      • FAQ
      • Troubleshooting
      • What’s New
        • Component Release Notes for Neuron Developer Tools
    • Neuron Profiler (archived)
      • Neuron Profiler User Guide (archived)
      • Neuron Profiler 2.0 (Beta) User Guide (archived)
    • Helper Tools
      • Check Model
      • GatherInfo
    • Transformers NeuronX (transformers-neuronx)
      • Setup
      • Developer Guide
        • Transformers NeuronX (transformers-neuronx) Developer Guide
        • Transformers NeuronX (transformers-neuronx) Developer Guide for Continuous Batching
      • Tutorials
        • Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1
        • Hugging Face facebook/opt-13b autoregressive sampling on Inf2 & Trn1
        • Hugging Face facebook/opt-30b autoregressive sampling on Inf2 & Trn1
        • Hugging Face facebook/opt-66b autoregressive sampling on Inf2
      • Misc
      • API Reference Guide
    • AWS Neuron Reference for NeMo Megatron
    • Apache MXNet Neuron — Archived
      • Apache MXNet Neuron Release Notes
      • MXNet Neuron Setup
      • Install MXNet Neuron
      • MXNet Neuron (“mxnet-neuron”) Setup on Amazon Linux 2
      • MXNet Neuron (“mxnet-neuron”) Setup on Amazon Linux 2
      • Update to latest MXNet Neuron
      • Install MXNet Neuron (Neuron 1.14.2)
      • Install MXNet Neuron (Neuron 1.15.0)
      • Install MXNet Neuron (Neuron 1.15.1)
      • Install MXNet Neuron (Neuron 1.15.2)
      • Install MXNet Neuron
      • Install MXNet Neuron
      • Install MXNet Neuron
      • Install MXNet Neuron
      • Inference (mxnet-neuron) (maintenance)
        • Tutorials
        • API Reference Guide
        • Developer Guide
        • Misc
      • Compile with Framework API and Deploy on EC2 Inf1 — Archived (MXNet)
      • Compile with SageMaker Neo and Deploy on SageMaker Hosting (Inf1) — Archived (MXNet)
      • MXNet Tutorial Setup
      • Tutorial: Neuron Apache MXNet Model Serving
    • TensorFlow Neuron
      • Tensorflow Neuron Setup
      • Get Started with TensorFlow Neuron (Archived)
      • Inference (Inf2 & Trn1)
        • Tutorials
        • API Reference Guide
        • Misc
      • Inference (Inf1)
        • Tutorials
        • Additional Examples
        • API Reference Guide
        • Misc
      • TensorFlow Neuron (Inf1)
        • TensorFlow 1.x (tensorflow-neuron) Compilation API
        • Install TensorFlow Neuron
        • Install previous TensorFlow Neuron releases
        • Install Previous Tensorflow Neuron Releases for Ubuntu (tensorflow-neuron)
        • Install Previous Tensorflow Neuron Releases for Ubuntu (tensorflow-neuron)
        • Install Previous Tensorflow Neuron Releases for Ubuntu (tensorflow-neuron)
        • Update to latest TensorFlow Neuron
        • Update to latest TensorFlow Neuron (tensorflow-neuron)
        • Update to latest TensorFlow Neuron (tensorflow-neuron)
        • Compile with Framework API and Deploy on EC2 Inf1 — Archived (TensorFlow)
        • Deploy Neuron Container on EC2 — Archived (TensorFlow)
        • Deploy Neuron Container on ECS — Archived (TensorFlow)
        • Deploy Neuron Container on EKS — Archived (TensorFlow)
        • Compile with SageMaker Neo and Deploy on SageMaker Hosting (Inf1) — Archived (TensorFlow)
        • TensorFlow 2.x FAQ
        • TensorFlow Tutorials
      • TensorFlow NeuronX (Inf2 & Trn1)
        • TensorFlow Setup Guide for Inf2 & Trn1
      • Setup (legacy Inf1)
    • PyTorch Neuron (torch-neuron) — Archived
      • API Reference Guide (torch-neuron)
        • PyTorch Neuron trace Python API
        • torch.neuron.DataParallel API
        • PyTorch Neuron (torch-neuron) Core Placement API
      • PyTorch-Neuron trace python API
      • PyTorch Neuron (torch-neuron) Core Placement API
      • torch.neuron.DataParallel API
      • Developer Guide (torch-neuron)
        • Running Inference on Variable Input Shapes with Bucketing
        • Data Parallel Inference on PyTorch Neuron
        • Developer Guide - PyTorch Neuron (torch-neuron) LSTM Support
        • PyTorch Neuron (torch-neuron) Core Placement
      • Troubleshooting Guide for PyTorch Neuron (torch-neuron)
      • Tutorials for Inference with torch-neuron (Inf1)
        • Computer Vision Tutorials
        • Natural Language Processing (NLP) Tutorials
        • Utilizing Neuron Capabilities Tutorials
      • Using NeuronCore Pipeline with PyTorch Tutorial
        • Using NeuronCore Pipeline with PyTorch
        • Using NeuronCore Pipeline with PyTorch
      • PyTorch Tutorial Setup
      • Setup Guide for Inf1
        • Fresh install
        • Update to latest release
        • Install previous releases
        • Install with support for cxx11 ABI
        • Install Previous PyTorch Neuron Releases for Amazon Linux (torch-neuron)
        • Update to latest PyTorch Neuron (torch-neuron)
        • Update to latest PyTorch Neuron (torch-neuron)
      • Install PyTorch Neuron (torch-neuron)
      • Update to latest PyTorch Neuron (torch-neuron)
      • Inference with torch-neuron (Inf1)
        • Tutorials
        • Additional Examples
        • API Reference Guide
        • Developer Guide
        • Misc
      • Additional Examples (torch-neuron)
        • AWS Neuron Samples GitHub Repository
      • Misc (torch-neuron)
        • PyTorch Neuron (torch-neuron) Supported operators
        • Troubleshooting Guide for PyTorch Neuron (torch-neuron)
        • Component Release Notes for Neuron PyTorch Framework
    • Bring Your Own Neuron Container to SageMaker Hosting (Inf1) — Archived
    • Bring Your Own Neuron Container to SageMaker Hosting (Inf2 or Trn1) — Archived
    • Deploy on Sagemaker Hosting
    • Compile with Sagemaker Neo and Deploy on Sagemaker Hosting (inf1)
    • Deploy Neuron Container on Kubernetes
    • Tutorial: Docker environment setup for Neuron Runtime 1.x
    • Archived setup guides
      • PyTorch Neuron setup for Rocky Linux 9 — Archived
      • Inf1 installation (legacy) — Archived
        • PyTorch on Inf1 (legacy) — Archived
      • Install previous PyTorch NeuronX releases for Ubuntu 20.04 — Archived
      • Install previous PyTorch NeuronX releases for Amazon Linux 2 — Archived
      • Install PyTorch NeuronX (Neuron 2.9.0) — Archived
      • Install PyTorch NeuronX (Neuron 2.8.0) — Archived
      • Install PyTorch NeuronX (Neuron 2.7.0) — Archived
    • NxD Training (archived)
      • Setup
      • YAML Configuration Settings
      • Neuronx Distributed Training Library Features
      • Developer Guide for Training with PyTorch NeuronX
      • How to debug models in PyTorch NeuronX
      • PyTorch NeuronX Environment Variables
      • PyTorch Neuron (torch-neuronx) for Training Troubleshooting Guide
      • Known Issues and Workarounds
      • CPU Mode Overview
      • Integrating a New Model
      • Integrating a new dataset/dataloader
      • Registering an optimizer and LR scheduler
      • NxD Training Compatibility with NeMo
      • Migrating from Neuron-NeMo-Megatron to Neuronx Distributed Training
      • Tensor Parallelism Overview
      • Pipeline Parallelism Overview
      • Context Parallelism Overview
      • Activation Memory Reduction
      • Analyze for Training Tutorial
      • Hugging Face BERT Pretraining Tutorial (Data-Parallel)
      • Multi-Layer Perceptron Training Tutorial
      • PyTorch Neuron for Trainium Hugging Face BERT MRPC task finetuning using Hugging Face Trainer API
      • ZeRO-1 Tutorial
      • Checkpoint Conversion
      • HuggingFace Llama3.1/Llama3-8B Pretraining
      • HuggingFace Llama3.1/Llama3-8B Supervised Fine-tuning
      • HuggingFace Llama3.1/Llama3-8B Efficient Supervised Fine-tuning with LoRA (Beta)
      • HF Llama3.1/Llama3-8B Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO) based Fine-tuning (Beta)
      • HuggingFace Llama3.1/Llama3-70B Pretraining
  • Repository
  • Suggest edit
  • Open issue
  • .rst

Transformers NeuronX Tutorials

Transformers NeuronX Tutorials#

  • Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1

  • Hugging Face facebook/opt-13b autoregressive sampling on Inf2 & Trn1

  • Hugging Face facebook/opt-30b autoregressive sampling on Inf2 & Trn1

  • Hugging Face facebook/opt-66b autoregressive sampling on Inf2

previous

Transformers NeuronX (transformers-neuronx) Developer Guide for Continuous Batching

next

Misc (transformers-neuronx)

By AWS

© Copyright 2026, Amazon.com.