NxD Inference#
Note
Maintenance status
NxD Inference and the inference-context NxD Core documentation in this section are in maintenance status — they continue to receive critical fixes and remain the supported path for Trn1/Inf2 inference and for custom NxD Inference API serving (for example Flux/diffusion). This content is not end-of-support. For new inference work, the go-forward path is the vLLM Neuron plugin; the task-oriented Inference on Neuron page shows where each path fits.
This section contains the technical documentation specific to the NxD Inference library included with the Neuron SDK.
What is NxD Inference?#
NxD Inference (NeuronX Distributed Inference) is an ML inference library included with the Neuron SDK that simplifies deploying deep learning models on AWS Inferentia and Trainium instances. It offers advanced features like continuous batching and speculative decoding for high-performance inference, and supports popular models like Llama-3.1, DBRX, and Mixtral.
With NxD Inference, developers can:
Deploy production-ready LLMs with minimal configuration
Leverage optimizations like KV Cache, Flash Attention, and Quantization
Distribute large models across multiple NeuronCores using Tensor and Sequence Parallelism
Integrate with vLLM for seamless production deployment
Customize and extend models with a modular design approach
With NxD Inference, developers can:
Use vLLM for Inference#
Neuron recommends that use vLLM when building your inference models. Read more about Neuron’s integration with vLLM here: vLLM on Neuron