NxD Inference

NxD Inference (where NxD stands for NeuronX Distributed) is an open-source PyTorch-based inference library that simplifies deep learning model deployment on AWS Inferentia and Trainium instances.

NxDI Inference Overview and Setup

• NxD Inference Overview

• NxD Inference Setup Guide

Get started with Inference

• Llama 3

Tutorials

• Tutorial: Deploying Llama3.1 405B (Trn2)

• Tutorial: Deploying Llama3.2 Multimodal Models

• Tutorial: Using Speculative Decoding to improve Llama-3.3-70B inference performance on Trn2 instances

• Tutorial: Scaling LLM Inference with Data Parallelism on Trn2

• Tutorial: Multi-LoRA serving for Llama-3.1-8B on Trn2 instances

• Tutorial: Using Speculative Decoding and Quantization to improve Llama-3.1-405B inference performance on Trn2 instances

• Tutorial: Evaluating Accuracy of Llama-3.1-70B on Neuron using open source datasets

• Tutorial: Disaggregated Inference [BETA]

• Tutorial: Static 1P1D Disaggregated Inference on Trn2 [BETA]

• Tutorial: Evaluating Performance of Llama-3.3-70B on Neuron using Performance CLI

Developer Guide

• NxD Inference Features Configuration Guide

• NxD Inference - Production Ready Models

• Onboarding models to run on NxD Inference

• vLLM User Guide for NxD Inference

• Testing modeling code with NxD Inference

• Migrating from NxD Core inference examples to NxD Inference

• Migrating from Transformers NeuronX to NeuronX Distributed(NxD) Inference

• LLM Inference Benchmarking guide

• Accuracy Evaluation of Models on Neuron Using Open Source Datasets

• Custom Quantization

• Disaggregated Inference [BETA]

• NxD Inference Weights Sharding Guide

• Evaluating Performance of Models on Neuron Using LLMPerf

API Reference Guide

• NxD Inference API Reference

App Notes

• Introducing NeuronX Distributed (NxD) Inference

• Parallelism Techniques for LLM Inference

Misc

• NxD Inference Release Notes (neuronx-distributed-inference)

• Troubleshooting Guide for NxD Inference