NxD Inference(Beta)

NxD Inference (where NxD stands for NeuronX Distributed) is an open-source PyTorch-based inference library that simplifies deep learning model deployment on AWS Inferentia and Trainium instances.

NxDI Inference Overview and Setup

• NxD Inference Overview

• NxD Inference Setup Guide

Developer Guide

• NxD Inference Features Configuration Guide

• NxD Inference - Production Ready Models

• Onboarding models to run on NxD Inference

• vLLM User Guide for NxD Inference

• Migrating from NxD Core inference examples to NxD Inference

• Migrating from Transformers NeuronX to NeuronX Distributed(NxD) Inference

API Reference Guide

• NxD Inference API Reference

Tutorials

• Tutorial: Deploying Llama3.1 405B (Trn2)

• Tutorial: Deploying Llama3.2 Multimodal Models

• Tutorial: Using Speculative Decoding to improve Llama-3.3-70B inference performance on Trn2 instances

App Notes

• Introducing NeuronX Distributed (NxD) Inference

Misc

• NxD Inference Release Notes (neuronx-distributed-inference)