Skip to main content
Ctrl+K
Neuron 2.32.0 is released! Check the What's New and Release Notes for more details.
Logo image
Ctrl+K
Search Engine: Default Google
  • About Neuron
    • Ask Amazon AI Helper Tools
    • Benchmarks
      • Inf1 Inference Performance
      • Inf2 Inference Performance
      • Trn1/Trn1n Inference Performance
      • Trn1/Trn1n Training Performance
    • Calculator
    • Beta Participation
    • Generative LLM Inference on Neuron
    • Model Samples
      • Training on Trn1
      • Inference on Inf2/Trn1/Trn2
      • Inference on Inf1
    • Neuron FAQ
    • Neuron Features
      • Custom C++ operators
      • Data types
      • Logical NeuronCore configuration
      • Neuron persistent cache
      • NeuronCore batching
      • NeuronCore pipeline
      • Rounding modes
    • Open Source
    • SDK Maintenance Policy
    • Security
    • Term Glossary
    • Troubleshooting
    • What is AWS Neuron?
    • Older Neuron FAQS
      • ONNX FAQ
      • Contributing Guidelines FAQ
      • Inference with Neuron FAQ
      • Troubleshooting for Inf1 FAQ
      • Training with Neuron FAQ
  • Neuron Architecture
    • AWS Inferentia
    • AWS Inferentia2
    • AWS Trainium
    • AWS Trainium2
    • AWS Trainium3
    • NeuronCore v1
    • NeuronCore v2
    • NeuronCore v3
    • NeuronCore v4
    • Inf1 Architecture
    • Inf2 Architecture
    • Trn1 Architecture
    • Trn2 Architecture
    • Trn3 Architecture
  • What's New
    • Release Notes
      • Neuron 2.32.0
        • NKI
        • NKI Library
        • Neuron Compiler
        • Neuron Runtime
        • Developer tools
        • Neuron Agentic Development
        • JAX NeuronX
        • vLLM Neuron
        • NxD Inference
        • Deep Learning AMIs
        • Deep Learning Containers
      • Neuron 2.31.1
      • Neuron 2.31.0
        • NKI
        • NKI Library
        • Neuron Compiler
        • Neuron Runtime
        • Developer tools
        • Neuron Agentic Development
        • PyTorch NeuronX
        • JAX NeuronX
        • NxD Core
        • NxD Inference
        • NxD Training
        • Deep Learning AMIs
        • Deep Learning Containers
      • Neuron 2.30.0
        • NKI
        • NKI Library
        • Neuron Compiler
        • Neuron Runtime
        • Developer tools
        • Neuron Agentic Development
        • Deep Learning AMIs
        • Deep Learning Containers
        • JAX
      • Previous Neuron versions
        • Neuron 2.31.0
        • Neuron 2.30.0
        • Neuron 2.29.1
        • Neuron 2.29.0
        • Neuron 2.28.1
        • Neuron 2.28.0
        • Neuron 2.27.1
        • Neuron 2.27.0
        • Neuron 2.26.1
        • Neuron 2.26.0
        • Neuron 2.25.0
      • Component release notes
        • Neuron Compiler
        • Neuron Agentic Development
        • Neuron Containers
        • Neuron Developer Tools
        • Neuron DLAMI
        • JAX NeuronX
        • NKI Library
        • Neuron Kernel Interface
        • NxD Core
        • NxD Inference
        • NxD Training
        • vLLM Neuron
        • PyTorch Neuron Framework
        • Neuron Runtime
        • Older components and features
      • Release artifacts
  • Announcements
  • News & Blogs
  • Contribute

Get Started

  • Quickstarts
    • Quickstart: Train a Model on Trainium
    • Quickstart: Run Inference on Inferentia
    • Quickstart: Serve models online with vLLM on Neuron
    • Quickstart: Run offline inference with vLLM on Neuron
    • Ask Amazon AI helper tools
    • Neuron GitHub Samples
  • Set Up Environments
    • PyTorch
      • New DLAMI
      • Update Existing DLAMI
      • New DLC
      • Update Existing DLC
      • New Manual Configuration
      • Update Manual Configuration
    • JAX
      • Install JAX via Deep Learning AMI
      • Install JAX via Deep Learning Container
      • Install JAX Manually
    • Multi-framework
    • Troubleshooting
  • Deploy Model Workloads
    • Deployment Overview
    • Pre-configured Environments
      • Deep Learning AMIs
      • DLC Images
      • Customize DLC
      • Docker Setup
      • Quickstart: Deploy DLC with vLLM
    • Use Amazon EC2
      • Train on EC2
      • Inference (Inf2, Trn1)
      • Inference with DLC
      • Inference (Inf1)
    • Use Amazon EKS
      • Set up EKS for Neuron
      • Components Included
      • Inference on EKS
      • Training on EKS
      • Dynamic Resource Allocation
        • Support Files
      • Neuron UltraServer Operator
      • UltraServer MPI Jobs
      • EKS Prerequisites
    • Use Amazon ECS
      • Inference on ECS
      • Training on ECS
      • Node Problem Detector
    • Use AWS Batch
      • Train on AWS Batch
    • Use AWS ParallelCluster
      • Train on ParallelCluster
    • Use Amazon SageMaker
      • Train on SageMaker
    • Infrastructure Components
      • Plugins Overview
      • Scheduler Extension
      • Scheduler Flow
      • Neuron Monitor
      • Node Problem Detector
      • NPD Permissions (IRSA)
    • Tutorials
      • Docker Environment Setup
      • Build and Run Containers
      • OCI Hook Setup
      • Inference Tutorials
        • Run Inference in PyTorch Neuron Container
        • Deploy a TensorFlow Resnet50 model as a Kubernetes service
      • Training Tutorials
        • Run Training in PyTorch Neuron Container
        • Deploy a simple mlp training script as a Kubernetes job
    • Docker Examples
      • Inference DLC Dockerfile
      • Libmode Dockerfile
      • TF Serving Dockerfile
      • TorchServe Config
      • Libmode Entrypoint
      • TorchServe Neuron
      • Training DLC Dockerfile
      • MLP Training Example
      • v1 App+RT Different Containers
      • v1 App+RT Same Container
      • v1 Neuron Runtime
      • v1 Torch Neuron
      • v1 Entrypoint
    • Third-party Solutions
    • FAQ
    • Troubleshooting
    • Container Release Notes
    • DLAMI Release Notes

ML Frameworks

  • ML Framework Support
  • PyTorch
    • About PyTorch on Neuron
    • Native PyTorch
    • PyTorch Setup
    • Inference
      • Tutorials
        • Compiling and Deploying HuggingFace Pretrained BERT on Trn1 or Inf2
        • BERT TorchServe Tutorial
        • LibTorch C++ Tutorial
        • Compiling and Deploying ResNet50 on Trn1 or Inf2
        • T5 model inference on Trn1 or Inf2
      • Additional Examples
        • AWS Neuron Samples GitHub Repository
        • Transformers Neuron GitHub samples
      • API Reference Guide
        • PyTorch NeuronX Tracing API for Inference
        • PyTorch Neuron (torch-neuronx) Weight Replacement API for Inference
        • PyTorch NeuronX NeuronCore Placement APIs
        • PyTorch NeuronX Analyze API for Inference
        • PyTorch NeuronX DataParallel API
      • Developer Guide
        • NeuronCore Allocation and Model Placement for Inference (torch-neuronx)
        • Comparison of Traced Inference versus XLA Lazy Tensor Inference (torch-neuronx)
      • Misc
        • Component Release Notes for Neuron PyTorch Framework
    • torch-neuron v. torch-neuronx
    • Release Notes
    • Supported operators
    • Profiling API
    • Profiling developer guide
    • neuron_parallel_compile CLI
    • Multi-node EFA setup
  • JAX
    • JAX NeuronX plugin Setup
    • JAX NeuronX Known Issues
    • API Reference Guide for JAX Neuronx
      • JAX NeuronX Environment Variables
    • Release Notes

Training

  • NxD Core (Training)
    • Setup
    • App Notes
      • Tensor Parallelism Overview
      • Pipeline Parallelism Overview
      • Activation Memory Reduction
      • Context Parallelism Overview
    • API Reference Guide
      • Distributed Strategies APIs
      • Training APIs
      • Inference APIs
      • ModelBuilderV2 API Reference
    • Developer Guide
      • Developer guide for Tensor Parallelism
      • Developer guide for Pipeline Parallelism
      • Developer guide for Activation Memory reduction
      • Developer guide for save/load checkpoint
      • Developer guide for Neuron-PT-Lightning
      • Developer guide for model and optimizer wrapper
      • Developer guide for LoRA finetuning
    • Tutorials
      • Training Tutorials
        • Training using Tensor Parallelism
        • Training Llama 3.1 8B/Llama 3 8B using TP and ZeRO-1
        • Training Llama 3.1 70B/Llama 3 70B using TP and PP
        • Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning
      • Inference Tutorials
        • [Broken] T5 inference with Tensor Parallelism
    • Misc
      • Component Release Notes for NxD Core

Inference

  • Inference on Neuron
  • NxD Inference + vLLM
    • Overview
    • Setup
    • vLLM
      • Quickstart: Offline Model Serving
      • Quickstart: Online Model Serving
      • vLLM on Neuron User Guide
      • Model Recipes
        • Llama 3.3 70B
        • Qwen3 235B A22B
      • Deploy Llama4 with vLLM
    • Tutorials
      • Disaggregated Inference (1P1D)
      • Disaggregated Inference
      • Flux Inference
      • Flux Inpainting
      • Benchmark using Performance CLI
      • GPT-OSS 120B
      • Llama3.1 405B on Trn2
      • Llama3.1 405B with Speculative Decoding
      • Llama3.1 70B Instruct Accuracy Evaluation
      • Llama3.1 8B with Multi-LoRA
      • Llama3.3 70B FP8
      • Llama3.3 70B with APC
      • Llama3.3 70B with Data Parallelism
      • Llama3.3 70B with Speculative Decoding
      • Llama4
      • Llama4 Legacy
      • Pixtral
      • Qwen3 MoE 235B
      • Qwen2 VL 7B
      • Speculative Decoding
      • Qwen3-VL 8B
    • Developer Guides
      • Accuracy Evaluation
      • Custom Quantization
      • Disaggregated Inference
      • Feature Guide
      • Using Pipeline Execution Mode
      • LLM Benchmarking
      • Migrate from TNX
      • Model Reference
      • MoE Architecture
      • Migrate from NxD Core
      • Onboarding Models
      • Performance Benchmarking CLI
      • vLLM Guide (Legacy)
      • vLLM Guide v1
      • Weights Sharding
      • Writing Tests
    • API Reference Guide
      • NxD Inference API Reference
    • App Notes
      • Parallelism Techniques for LLM Inference
    • Release Notes
    • Misc
      • Component Release Notes for NxD Inference
      • Troubleshooting Guide for NxD Inference
  • NxD Core (Inference)
    • Setup
    • App Notes
      • Tensor Parallelism Overview
      • Pipeline Parallelism Overview
      • Activation Memory Reduction
      • Context Parallelism Overview
    • API Reference Guide
      • Distributed Strategies APIs
      • Training APIs
      • Inference APIs
      • ModelBuilderV2 API Reference
    • Developer Guide
      • About NeuronX-Distributed (NxD) Inference
    • LoRA Guide
    • Tutorials
      • Training Tutorials
        • Training using Tensor Parallelism
        • Training Llama 3.1 8B/Llama 3 8B using TP and ZeRO-1
        • Training Llama 3.1 70B/Llama 3 70B using TP and PP
        • Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning
      • Inference Tutorials
        • [Broken] T5 inference with Tensor Parallelism
    • Misc
      • Component Release Notes for NxD Core
  • vLLM Neuron (Beta)
    • Get started
      • Setting up
      • Online serving quickstart
      • Offline serving quickstart
      • Migrating from NxDI
    • Deploy & serve
      • Features guide
      • Configuration reference
      • Profiling workloads
      • Feature–model compatibility
    • Model Recipes
      • Llama 3
      • GPT-OSS
      • Qwen3-VL
      • Qwen3-Embedding
    • Tutorials
      • Deploying Llama 3.3 70B
      • EAGLE3 speculative decoding (Llama 3.1)
      • EAGLE3 speculative decoding (GPT-OSS)
      • Disaggregated inference (1P1D and xPyD)
      • Deploying gpt-oss
      • Benchmarking prefix caching (GPT-OSS)
      • Deploying Qwen3-VL-32B
      • Disaggregated encoder (1E1PD and xEyPD)
      • Deploying Qwen3-Embedding-8B
    • Model development
      • Onboarding a model
      • Onboarding a vision-language model
      • Optimizing a vision-language model
      • CPU development workflow
      • NKI CPU simulator
      • Debugging model code
      • Debugging accuracy issues
      • Accuracy debugger tools
    • Concepts & architecture
      • Parallelism
        • Attention DP
        • Component-Level DP Sharding
        • Data Parallelism (DP)
        • DCP (Decode Context Parallelism)
        • Expert Parallelism (EP)
        • Tensor Parallelism (TP)
        • Vision Encoder Parallelism
      • Speculation
        • Speculative Decoding (EAGLE3) in the vLLM Neuron Framework
      • Multimodal
        • Block Packing Vision Attention
        • On-Device Encoder Cache
        • M-RoPE (Multimodal Rotary Position Embedding)
      • vLLM integration
        • vLLM Integration Design Reference
        • KV Caching with vLLM
        • Additional Configuration Options
        • Async Scheduling and Async Execution
        • Decode Context-Length Bucketing
        • determine_available_memory design
        • KV Cache Quantization
        • vLLM Metrics Integration
        • Runtime NRT Profiling via vLLM’s Profiler Interface
        • Scheduler Design
        • Padding and Batching with Block KV
        • Prefill Processing
        • Pooling models on Trainium
        • Prompt embeddings on Neuron
        • Structured Outputs and Tool Calling
        • Disaggregated Inference — Design Document
      • Framework
        • Asynchronous Execution Framework Capability
        • Checkpoint Loading Framework Capability
        • Canonical Model — Design & Porting Guide
        • Scalable Model Registry Design
      • Compilation
        • Compilation Cache
        • CPU Compilation
        • FX Passes Architecture Design
        • Aliasing Output Rewrite Pass
        • FX Device Rewriting for XLA Compilation
        • In-Place to Out-of-Place Rewrite Pass
      • Accuracy
        • Accuracy Debugging Design
        • Dataset Evaluation Design
        • KV Cache Analysis Design
        • vLLM Neuron Logit Validation Design
        • Module-Level Accuracy Test Guidelines
        • Tensor Capture Design
        • Tensor Compare Design
        • Tensor Replacement — Design
        • Input Snapshot Design
  • vLLM Omni Neuron (Beta)
    • Get started
      • Setup guide
      • Online serving quickstart
      • Offline serving quickstart
    • Guides
      • Features guide
    • Model recipes
      • WAN 2.2 T2V
      • WAN 2.2 I2V
    • Tutorials
      • Deploy Wan2.2-A14B
    • Model development
      • Onboarding a model
      • Accuracy evaluation and debugging
      • Kernel implementations
        • Const-max ring attention
        • Fused adaptive LayerNorm and FP8 quantization
        • QKV projection with QK Distributed RMSNorm and RoPE fusion
      • Optimizing offline video generation
    • Concepts & architecture
      • vLLM Omni Neuron overview
      • Context parallelism

Developer Tools

  • Neuron Developer Tools
    • System Tools
      • Neuron-Monitor User Guide
      • Neuron-Top User Guide
      • Neuron-LS User Guide
      • Neuron-Sysfs User Guide
      • NCCOM-TEST User Guide
    • Third-party Tools
    • Tutorials
      • Profiling a vLLM Inference Workload on AWS Trainium
      • Track System Resource Utilization during Training with neuron-monitor using PyTorch Neuron
    • Release Notes
  • Neuron Agentic Development
    • Getting Started
    • Developer Guides
      • Autoport NxDI Deep Dive
      • Autoport vLLM Neuron Deep Dive
      • Equivalence Validation Deep Dive
    • Tutorials
      • Port a model with Autoport
      • Port a model to vLLM Neuron
      • Validate a port with Equivalence
  • Neuron Explorer
    • Get Started
    • Capture Profiles
    • UI Overview
    • Device Trace Viewer
    • System Trace Viewer
    • Hierarchy Viewer
    • Source Code Viewer
    • Summary Viewer
    • Tensor Viewer
    • Memory Viewer
    • Region Highlighter
    • AI Recommendation Viewer
    • Profile Parquet Schema Reference
    • Analyze Profile Output
    • View Profiles with Perfetto
    • Neuron Profiler to Neuron Explorer Migration Guide
    • Troubleshooting & FAQs
    • Environment Variables
    • Glossary

Compiler, Runtime, & Collectives

  • Graph Compiler
    • About neuronx-cc
    • NeuronX Compiler for Trn* & Inf2
      • API Reference Guide
      • How-to: Generate a NEFF file
      • How-to: Convolution
        • * `UNet training sample <https://github.com/aws-neuron/aws-neuron-samples/tree/master/torch-neuronx/training/unet_image_segmentation>`_ - Sample UNet training implementation
      • FAQ
    • Error Codes
      • NCC_EARG001
      • NCC_EBIR023
      • NCC_EBVF030
      • NCC_EHCA005
      • NCC_EMOD025
      • NCC_EOOM001
      • NCC_EOOM002
      • NCC_ESFH002
      • NCC_ESPP004
      • NCC_ESPP047
      • NCC_EUOC002
      • NCC_EVRF001
      • NCC_EVRF004
      • NCC_EVRF005
      • NCC_EVRF006
      • NCC_EVRF007
      • NCC_EVRF009
      • NCC_EVRF010
      • NCC_EVRF011
      • NCC_EVRF013
      • NCC_EVRF015
      • NCC_EVRF016
      • NCC_EVRF017
      • NCC_EVRF018
      • NCC_EVRF019
      • NCC_EVRF022
      • NCC_EVRF031
      • NCC_EVRF035
      • NCC_EVRF036
      • NCC_EVRF037
      • NCC_EVRF038
      • NCC_EVRF039
      • NCC_EVRF040
      • NCC_EVRF041
      • NCC_EVRF042
      • NCC_EVRF043
      • NCC_EVRF044
      • NCC_EVRF045
      • NCC_EVRF046
      • NCC_EVRF047
      • NCC_EVRF048
      • NCC_EVRF049
      • NCC_EVRF050
      • NCC_EVRF051
      • NCC_EVRF052
      • NCC_EVRF053
      • NCC_EVRF054
      • NCC_EVRF055
      • NCC_EVRF056
      • NCC_EVRF057
      • NCC_EVRF058
      • NCC_EVRF059
      • NCC_EVRF060
      • NCC_EVRF061
      • NCC_EVRF062
      • NCC_EVRF063
      • NCC_EVRF064
      • NCC_EVRF065
      • NCC_EXSP001
      • NCC_EXTP004
    • Release Notes
    • neuron-cc (Legacy)
      • API Reference Guide
        • Neuron compiler CLI Reference Guide (neuron-cc)
      • CLI Reference
      • Developer Guide
      • FAQ
  • Neuron Runtime
    • Overview
    • Deep Dives
      • Compute-Communication Overlap
      • Neuron Device Memory
      • Direct HBM Tensor Allocation
      • Runtime Performance Tips
      • Neuron Runtime Core Dumps
      • Inter-node Collectives
      • Intra-node Collectives
    • How-To Guides
      • Developer guide
      • Migrate to the explicit async APIs
      • Configuration guide
    • API Reference
      • NRT API
      • NRT Status
      • NRT Version
      • Debug Stream APIs
      • NRT Async APIs Overview
      • NRT Async APIs Best Practices
      • NRT Async APIs Examples
      • NRT Async
      • NRT Async Send/Recv
      • NRT Profile
      • NRT System Trace
      • Debug Stream
      • NEC API
      • NDL API
      • Neuron Driver Shared
      • Tensor Batch Operations
      • Neuron Datastore
      • NRT Experimental
    • nrtpy (Neuron Runtime Python)
      • Getting started with nrtpy
      • Test and debug a NKI kernel end-to-end
      • Validate, benchmark, and trace a kernel
      • Troubleshooting
      • Model API
      • Tensor API
      • Configuration
      • Error handling
    • NRT Debug Stream
    • Troubleshooting on Inf1 and Trn1
    • Release Notes
    • FAQ
  • Collectives
  • Neuron C++ Custom Operators
    • API Reference Guide
      • Custom Operators API Reference Guide [Beta]
    • Developer Guide
      • Neuron Custom C++ Operators Developer Guide [Beta]
    • Tutorials
      • Neuron Custom C++ Operators in MLP Training
      • Neuron Custom C++ Operators Performance Optimization
    • Misc (Neuron Custom C++ Operators)
      • Neuron Custom C++ Tools Release Notes
      • Neuron Custom C++ Library Release Notes

Neuron Kernel Interface (NKI)

  • NKI
    • Get started
      • Environment Setup
      • First NKI Kernel
      • NKI Language Guide
      • Concepts
        • Memory Hierarchy
        • Data Representation
        • Indexing
        • Tiling
        • Direct Memory Access
        • Logical Neuron Cores
    • Guides
      • Tutorials
        • Matrix Multiplication
        • AveragePool2D
        • Transpose2D
        • Fused Mamba
        • Introduction to NKI Kernel Optimization
      • Architecture
        • Trainium/Inferentia2 Guide
        • Trainium2 Guide
        • Trainium3 Guide
      • NKI CPU Simulator
      • Insert NKI Kernels into Models
      • Use NKI Scheduling APIs
      • Profile a NKI Kernel
    • Deep dives
      • Performance Optimizations
      • MXFP8/4 Matrix Multiplication
      • NKI Access Patterns
      • NKI Dynamic Loops
      • Descriptor Generation Engine (DGE)
      • DMA Bandwidth Guide
      • About the NKI Compiler
    • Migration
    • NKI FAQ
  • NKI API Reference
    • nki
      • nki.jit
      • nki.simulate
    • nki.isa
      • nki.isa.nc_matmul
      • nki.isa.nc_matmul_mx
      • nki.isa.nc_transpose
      • nki.isa.activation
      • nki.isa.activation_reduce
      • nki.isa.activate2
      • nki.isa.tensor_reduce
      • nki.isa.tensor_partition_reduce
      • nki.isa.tensor_tensor
      • nki.isa.tensor_tensor_scan
      • nki.isa.scalar_tensor_tensor
      • nki.isa.tensor_scalar
      • nki.isa.tensor_scalar_reduce
      • nki.isa.tensor_scalar_cumulative
      • nki.isa.tensor_copy
      • nki.isa.tensor_copy_predicated
      • nki.isa.exponential
      • nki.isa.reciprocal
      • nki.isa.quantize_mx
      • nki.isa.iota
      • nki.isa.dropout
      • nki.isa.affine_select
      • nki.isa.range_select
      • nki.isa.select_reduce
      • nki.isa.sequence_bounds
      • nki.isa.memset
      • nki.isa.bn_stats
      • nki.isa.bn_aggr
      • nki.isa.local_gather
      • nki.isa.nc_n_gather
      • nki.isa.dma_copy
      • nki.isa.dma_transpose
      • nki.isa.dma_compute
      • nki.isa.max8
      • nki.isa.nc_find_index8
      • nki.isa.nc_match_replace8
      • nki.isa.nc_stream_shuffle
      • nki.isa.register_alloc
      • nki.isa.register_load
      • nki.isa.register_move
      • nki.isa.register_store
      • nki.isa.core_barrier
      • nki.isa.sendrecv
      • nki.isa.rng
      • nki.isa.rand2
      • nki.isa.rand_set_state
      • nki.isa.rand_get_state
      • nki.isa.set_rng_seed
      • nki.isa.nonzero_with_count
      • nki.isa.topk
      • nki.isa.engine
      • nki.isa.dma_engine
      • nki.isa.reduce_cmd
      • nki.isa.dge_mode
      • nki.isa.oob_mode
      • nki.isa.matmul_perf_mode
      • nki.isa.nc_version
      • nki.isa.get_nc_version
      • nki.isa.VirtualRegister
      • nki.isa.NkiInstruction
    • nki.language
      • nki.language.ndarray
      • nki.language.zeros
      • nki.language.ones
      • nki.language.full
      • nki.language.zeros_like
      • nki.language.empty_like
      • nki.language.shared_constant
      • nki.language.shared_identity_matrix
      • nki.language.rand
      • nki.language.random_seed
      • nki.language.NkiTensor
      • nki.language.load
      • nki.language.load_transpose2d
      • nki.language.store
      • nki.language.copy
      • nki.language.matmul
      • nki.language.transpose
      • nki.language.abs
      • nki.language.abs_max
      • nki.language.abs_min
      • nki.language.add
      • nki.language.arctan
      • nki.language.ceil
      • nki.language.cos
      • nki.language.exp
      • nki.language.floor
      • nki.language.log
      • nki.language.maximum
      • nki.language.minimum
      • nki.language.multiply
      • nki.language.negative
      • nki.language.power
      • nki.language.reciprocal
      • nki.language.rsqrt
      • nki.language.sign
      • nki.language.sin
      • nki.language.sqrt
      • nki.language.square
      • nki.language.subtract
      • nki.language.tan
      • nki.language.tanh
      • nki.language.trunc
      • nki.language.relu
      • nki.language.sigmoid
      • nki.language.silu
      • nki.language.silu_dx
      • nki.language.gelu
      • nki.language.gelu_dx
      • nki.language.gelu_apprx_sigmoid
      • nki.language.gelu_apprx_sigmoid_dx
      • nki.language.gelu_apprx_tanh
      • nki.language.mish
      • nki.language.softplus
      • nki.language.softmax
      • nki.language.erf
      • nki.language.erf_dx
      • nki.language.dropout
      • nki.language.rms_norm
      • nki.language.all
      • nki.language.max
      • nki.language.mean
      • nki.language.min
      • nki.language.prod
      • nki.language.sum
      • nki.language.var
      • nki.language.equal
      • nki.language.not_equal
      • nki.language.less
      • nki.language.less_equal
      • nki.language.greater
      • nki.language.greater_equal
      • nki.language.logical_and
      • nki.language.logical_or
      • nki.language.logical_xor
      • nki.language.logical_not
      • nki.language.bitwise_and
      • nki.language.bitwise_or
      • nki.language.bitwise_xor
      • nki.language.invert
      • nki.language.left_shift
      • nki.language.right_shift
      • nki.language.broadcast_to
      • nki.language.ds
      • nki.language.expand_dims
      • nki.language.where
      • nki.language.gather_flattened
      • nki.language.affine_range
      • nki.language.dynamic_range
      • nki.language.sequential_range
      • nki.language.static_range
      • nki.language.fori_loop
      • nki.language.while_loop
      • nki.language.psum
      • nki.language.sbuf
      • nki.language.hbm
      • nki.language.private_hbm
      • nki.language.shared_hbm
      • nki.language.is_psum
      • nki.language.is_sbuf
      • nki.language.is_hbm
      • nki.language.is_on_chip
      • nki.language.device_print
      • nki.language.no_reorder
      • nki.language.program_id
      • nki.language.num_programs
      • nki.language.program_ndim
      • nki.language.bool_
      • nki.language.int8
      • nki.language.int16
      • nki.language.int32
      • nki.language.uint8
      • nki.language.uint16
      • nki.language.uint32
      • nki.language.float16
      • nki.language.float32
      • nki.language.bfloat16
      • nki.language.tfloat32
      • nki.language.float8_e4m3
      • nki.language.float8_e5m2
      • nki.language.float8_e4m3fn
      • nki.language.float8_e8m0fnu
      • nki.language.float8_e5m2_x4
      • nki.language.float8_e4m3fn_x4
      • nki.language.float4_e2m1fn_x4
      • nki.language.tile_size
      • nki.language.prelu
      • nki.language.bypass
    • nki.collectives
      • nki.collectives.all_reduce
      • nki.collectives.all_gather
      • nki.collectives.all_gather_v
      • nki.collectives.reduce_scatter
      • nki.collectives.all_to_all
      • nki.collectives.all_to_all_v
      • nki.collectives.collective_permute
      • nki.collectives.collective_permute_implicit
      • nki.collectives.collective_permute_implicit_reduce
      • nki.collectives.collective_permute_implicit_current_processing_rank_id
      • nki.collectives.rank_id
      • nki.collectives.ReplicaGroup
    • NKI ISA API Common Fields
  • NKI Library
    • Get started
    • Overview
    • Kernel Design Specs
      • RMSNorm-Quant
    • Kernel API Reference
      • Argsort Unstable
      • Attention Block TKG
      • Attention CTE
      • Attention KV-Parallel Segmented CTE
      • Attention MXFP8 TKG
      • Attention Segmented CTE
      • Attention TKG
      • Blockwise MM Backward
      • Blockwise MM Backward MXFP8
      • Blockwise MM Forward MXFP8
      • Blockwise MM Shard-on-Block
      • Build All-to-All-V Metadata
      • Build All2All Dispatch Metadata
      • Collective Communication Kernels
      • Conv1D
      • Conv3D
      • Conv3D Temporal Unroll
      • Conv3D Transpose
      • Cross Entropy
      • Cumsum
      • Depthwise Conv1D
      • Dynamic Elementwise Add
      • FGCC
      • Find Nonzero Indices
      • Fine-Grained All-Gather
      • Foreach Elementwise
      • Foreach Norm
      • FP8 Quantize
      • Fused Adam
      • Gather
      • GPSIMD Top-K
      • Linear Scan
      • Matmul MXFP8
      • Matmul MXFP8 Backward
      • MLA QKV CTE
      • MLA Sparse Attention CTE
      • MLA V-Up O-Proj CTE
      • MLP
      • MLP Backward MXFP8
      • MLP Forward MXFP8
      • MoE CTE
      • MoE TKG
      • MS Deformable Attention
      • MS Deformable Attention Backward
      • MX MoE Block TKG Wrapper
      • Output Projection CTE
      • Output Projection TKG
      • Pad
      • Permute A2AV
      • Permute Routed Tokens
      • QKV
      • QKV Batch Shard
      • QKV CTE MLA
      • Quantize MXFP8
      • Ring Attention Backward
      • Ring Attention Forward
      • Ring Attention Unpermute
      • RMSNorm-Quant
      • RMSNorm MX Prefill
      • RMSNorm Router Top-K A2AV
      • RMSNorm Router Top-K TKG
      • RNG
      • RoPE
      • Router Top-K
      • SBUF-to-SBUF All-Gather
      • Scatter-Add
      • Selective Scan
      • Sparse Attention Indexer MX BF16Score
      • SSD
      • SWA Fused CTE
      • Top-K Reduce
      • Transformer TKG
      • Unpermute A2AV
    • Kernel Utilities
      • SbufManager (Allocator)
      • TensorView
    • Release Notes

Archive

  • Archived content
    • Fine-tune T5 model on Trn1
    • Running SSD300 with AWS Neuron
    • Megatron GPT Pretraining
    • Launch a GPT-3 pretraining job using neuronx-nemo-megatron
    • Training GPT-NeoX 20B with Tensor Parallelism and ZeRO-1 Optimizer
    • Fine-tuning Llama2 7B with tensor parallelism and ZeRO-1 optimizer using Neuron PyTorch-Lightning
    • Training Llama-2-7B/13B/70B using Tensor Parallelism and Pipeline Parallelism with Neuron PyTorch-Lightning
    • Training CodeGen2.5 7B with Tensor Parallelism and ZeRO-1 Optimizer
    • Profiling Multi-Node Training Jobs with Neuron Explorer
    • Training GPT-NeoX 6.9B with Tensor Parallelism and ZeRO-1 Optimizer
    • Neuron application notes
      • Neuron Runtime Library
      • Mixed precision (neuron-cc)
      • Mixed precision (neuronx-cc)
      • Introducing NxD Inference
      • Introducing NxD Training
      • Performance
      • Parallel execution
      • PyTorch for Neuron
        • Running inference on variable input shapes with bucketing
        • Running R-CNNs on Inf1
        • Data Parallel Inference on Torch Neuron
      • PyTorch for NeuronX
        • Introducing PyTorch 2.6 Support
        • Introducing PyTorch 2.7 Support
        • Introducing PyTorch 2.8 Support
        • Introducing PyTorch 2.9 Support
        • Introducing PyTorch 2.5 Support
        • Migration From XLA_USE_BF16/XLA_DOWNCAST_BF16
        • Data Parallel Inference on torch_neuronx
        • Graph Partitioner on torch_neuronx
    • Neuron Plugin for TensorBoard (Inf1)
    • TensorBoard plugin (archived)
      • NeuronX Plugin for TensorBoard (Trn1) (archived)
      • Neuron Plugin for TensorBoard (Inf1)
      • Profiling PyTorch NeuronX with TensorBoard (archived)
      • Track Training Progress in TensorBoard using PyTorch Neuron (archived)
    • NeuronPerf (Beta)
      • NeuronPerf Install
      • Overview
      • Terminology
      • Examples
      • Benchmark Guide
      • Evaluate Guide
      • Compile Guide
      • Model Index Guide
      • API
      • Framework Notes
      • FAQ
      • Troubleshooting
      • What’s New
        • Component Release Notes for Neuron Developer Tools
    • Neuron Profiler (archived)
      • Neuron Profiler User Guide (archived)
      • Neuron Profiler 2.0 (Beta) User Guide (archived)
    • Helper Tools
      • Check Model
      • GatherInfo
    • Transformers NeuronX (transformers-neuronx)
      • Setup
      • Developer Guide
        • Transformers NeuronX (transformers-neuronx) Developer Guide
        • Transformers NeuronX (transformers-neuronx) Developer Guide for Continuous Batching
      • Tutorials
        • Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1
        • Hugging Face facebook/opt-13b autoregressive sampling on Inf2 & Trn1
        • Hugging Face facebook/opt-30b autoregressive sampling on Inf2 & Trn1
        • Hugging Face facebook/opt-66b autoregressive sampling on Inf2
      • Misc
      • API Reference Guide
    • AWS Neuron Reference for NeMo Megatron
    • Apache MXNet Neuron — Archived
      • Apache MXNet Neuron Release Notes
      • MXNet Neuron Setup
      • Install MXNet Neuron
      • MXNet Neuron (“mxnet-neuron”) Setup on Amazon Linux 2
      • MXNet Neuron (“mxnet-neuron”) Setup on Amazon Linux 2
      • Update to latest MXNet Neuron
      • Install MXNet Neuron (Neuron 1.14.2)
      • Install MXNet Neuron (Neuron 1.15.0)
      • Install MXNet Neuron (Neuron 1.15.1)
      • Install MXNet Neuron (Neuron 1.15.2)
      • Install MXNet Neuron
      • Install MXNet Neuron
      • Install MXNet Neuron
      • Install MXNet Neuron
      • Inference (mxnet-neuron) (maintenance)
        • Tutorials
        • API Reference Guide
        • Developer Guide
        • Misc
      • Compile with Framework API and Deploy on EC2 Inf1 — Archived (MXNet)
      • Compile with SageMaker Neo and Deploy on SageMaker Hosting (Inf1) — Archived (MXNet)
      • MXNet Tutorial Setup
      • Tutorial: Neuron Apache MXNet Model Serving
    • TensorFlow Neuron
      • Tensorflow Neuron Setup
      • Get Started with TensorFlow Neuron (Archived)
      • Inference (Inf2 & Trn1)
        • Tutorials
        • API Reference Guide
        • Misc
      • Inference (Inf1)
        • Tutorials
        • Additional Examples
        • API Reference Guide
        • Misc
      • TensorFlow Neuron (Inf1)
        • TensorFlow 1.x (tensorflow-neuron) Compilation API
        • Install TensorFlow Neuron
        • Install previous TensorFlow Neuron releases
        • Install Previous Tensorflow Neuron Releases for Ubuntu (tensorflow-neuron)
        • Install Previous Tensorflow Neuron Releases for Ubuntu (tensorflow-neuron)
        • Install Previous Tensorflow Neuron Releases for Ubuntu (tensorflow-neuron)
        • Update to latest TensorFlow Neuron
        • Update to latest TensorFlow Neuron (tensorflow-neuron)
        • Update to latest TensorFlow Neuron (tensorflow-neuron)
        • Compile with Framework API and Deploy on EC2 Inf1 — Archived (TensorFlow)
        • Deploy Neuron Container on EC2 — Archived (TensorFlow)
        • Deploy Neuron Container on ECS — Archived (TensorFlow)
        • Deploy Neuron Container on EKS — Archived (TensorFlow)
        • Compile with SageMaker Neo and Deploy on SageMaker Hosting (Inf1) — Archived (TensorFlow)
        • TensorFlow 2.x FAQ
        • TensorFlow Tutorials
      • TensorFlow NeuronX (Inf2 & Trn1)
        • TensorFlow Setup Guide for Inf2 & Trn1
      • Setup (legacy Inf1)
    • PyTorch Neuron (torch-neuron) — Archived
      • API Reference Guide (torch-neuron)
        • PyTorch Neuron trace Python API
        • torch.neuron.DataParallel API
        • PyTorch Neuron (torch-neuron) Core Placement API
      • PyTorch-Neuron trace python API
      • PyTorch Neuron (torch-neuron) Core Placement API
      • torch.neuron.DataParallel API
      • Developer Guide (torch-neuron)
        • Running Inference on Variable Input Shapes with Bucketing
        • Data Parallel Inference on PyTorch Neuron
        • Developer Guide - PyTorch Neuron (torch-neuron) LSTM Support
        • PyTorch Neuron (torch-neuron) Core Placement
      • Troubleshooting Guide for PyTorch Neuron (torch-neuron)
      • Tutorials for Inference with torch-neuron (Inf1)
        • Computer Vision Tutorials
        • Natural Language Processing (NLP) Tutorials
        • Utilizing Neuron Capabilities Tutorials
      • Using NeuronCore Pipeline with PyTorch Tutorial
        • Using NeuronCore Pipeline with PyTorch
        • Using NeuronCore Pipeline with PyTorch
      • PyTorch Tutorial Setup
      • Setup Guide for Inf1
        • Fresh install
        • Update to latest release
        • Install previous releases
        • Install with support for cxx11 ABI
        • Install Previous PyTorch Neuron Releases for Amazon Linux (torch-neuron)
        • Update to latest PyTorch Neuron (torch-neuron)
        • Update to latest PyTorch Neuron (torch-neuron)
      • Install PyTorch Neuron (torch-neuron)
      • Update to latest PyTorch Neuron (torch-neuron)
      • Inference with torch-neuron (Inf1)
        • Tutorials
        • Additional Examples
        • API Reference Guide
        • Developer Guide
        • Misc
      • Additional Examples (torch-neuron)
        • AWS Neuron Samples GitHub Repository
      • Misc (torch-neuron)
        • PyTorch Neuron (torch-neuron) Supported operators
        • Troubleshooting Guide for PyTorch Neuron (torch-neuron)
        • Component Release Notes for Neuron PyTorch Framework
    • Bring Your Own Neuron Container to SageMaker Hosting (Inf1) — Archived
    • Bring Your Own Neuron Container to SageMaker Hosting (Inf2 or Trn1) — Archived
    • Deploy on Sagemaker Hosting
    • Compile with Sagemaker Neo and Deploy on Sagemaker Hosting (inf1)
    • Deploy Neuron Container on Kubernetes
    • Tutorial: Docker environment setup for Neuron Runtime 1.x
    • Archived setup guides
      • PyTorch Neuron setup for Rocky Linux 9 — Archived
      • Inf1 installation (legacy) — Archived
        • PyTorch on Inf1 (legacy) — Archived
      • Install previous PyTorch NeuronX releases for Ubuntu 20.04 — Archived
      • Install previous PyTorch NeuronX releases for Amazon Linux 2 — Archived
      • Install PyTorch NeuronX (Neuron 2.9.0) — Archived
      • Install PyTorch NeuronX (Neuron 2.8.0) — Archived
      • Install PyTorch NeuronX (Neuron 2.7.0) — Archived
    • NxD Training (archived)
      • Setup
      • YAML Configuration Settings
      • Neuronx Distributed Training Library Features
      • Developer Guide for Training with PyTorch NeuronX
      • How to debug models in PyTorch NeuronX
      • PyTorch NeuronX Environment Variables
      • PyTorch Neuron (torch-neuronx) for Training Troubleshooting Guide
      • Known Issues and Workarounds
      • CPU Mode Overview
      • Integrating a New Model
      • Integrating a new dataset/dataloader
      • Registering an optimizer and LR scheduler
      • NxD Training Compatibility with NeMo
      • Migrating from Neuron-NeMo-Megatron to Neuronx Distributed Training
      • Tensor Parallelism Overview
      • Pipeline Parallelism Overview
      • Context Parallelism Overview
      • Activation Memory Reduction
      • Analyze for Training Tutorial
      • Hugging Face BERT Pretraining Tutorial (Data-Parallel)
      • Multi-Layer Perceptron Training Tutorial
      • PyTorch Neuron for Trainium Hugging Face BERT MRPC task finetuning using Hugging Face Trainer API
      • ZeRO-1 Tutorial
      • Checkpoint Conversion
      • HuggingFace Llama3.1/Llama3-8B Pretraining
      • HuggingFace Llama3.1/Llama3-8B Supervised Fine-tuning
      • HuggingFace Llama3.1/Llama3-8B Efficient Supervised Fine-tuning with LoRA (Beta)
      • HF Llama3.1/Llama3-8B Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO) based Fine-tuning (Beta)
      • HuggingFace Llama3.1/Llama3-70B Pretraining
  • Repository
  • Open issue

Posts tagged announce-eos-tensorflow-tutorial

Announcing End of Support for Tensorflow Neuron Inf1 SSD300 tutorial starting next release

  • 24 June 2025
  • en
  • announce-eos-tensorflow-tutorial

Neuron release 2.24 will be the last release that will include support for Tensorflow Neuron Inf1 SSD300 tutorial. Future Neuron releases will not include support for Tensorflow Neuron Inf1 SSD300 tutorial due to security issues.

Read more ...

By AWS

© Copyright 2026, Amazon.com.