Legacy inference frameworks#

Note

End of support

NxD Inference and the inference-context NxD Core documentation in this section are legacy and are announced end of support on Trn2 and Trn3 starting the Neuron 2.34 release. For new inference work, the recommended path is the vLLM Neuron plugin; the task-oriented Inference on Neuron page shows where each path fits.

This section is the home for the legacy NxD Inference (NxDI) and inference-context NxD Core documentation on the AWS Neuron SDK. It applies to developers who are still running NxDI-based inference on Trn1/Inf2 instances or building custom serving on the NxD Inference API, while vLLM Neuron becomes the recommended path.

The content here is live, indexed, and searchable — it is not archived. It is kept together as a single reference section so that inference content that is legacy but no longer recommended for new work stays discoverable in one place.

Applies to#

This section is applicable to:

  • Inference on Trn1 and Inf2 using NxD Inference (NxDI).

  • Custom serving built on the NxD Inference API, such as diffusion and Flux models.

  • Inference-context NxD Core primitives used by NxDI.

What lives here#

  • NxD Inference (NxDI) — the legacy inference library: setup, model support, developer guides, tutorials, and API references.

  • NxD Core (inference context) — the inference-context core primitives, developer guides, and tutorials that NxDI builds on.

Where to go for new work#

For new inference projects, start from the recommended path rather than this section:

  • The vLLM Neuron plugin is the recommended top-level inference section.

  • The task-oriented Inference on Neuron overview shows where each serving path fits, including when NxD Inference remains the right choice.