This document is relevant for: Inf2, Trn1, Trn1n

Inference with torch-neuronx (Inf2 & Trn1/Trn1n)

Setup (torch-neuronx)

Tutorials (torch-neuronx)

• HuggingFace pretrained BERT tutorial [html] [notebook]

• TorchServe tutorial [html]

• LibTorch C++ tutorial (for torch-neuron and torch-neuronx) [html]

• Torchvision ResNet50 tutorial [html] [notebook]

• T5 inference tutorial [html] [notebook]

Note

To use Jupyter Notebook see:

• Jupyter Notebook QuickStart

• Running Jupyter Notebook as script

Additional Examples (torch-neuronx)

• AWS Neuron Samples GitHub Repository

• Transformers Neuron GitHub samples

API Reference Guide (torch-neuronx)

• PyTorch NeuronX Tracing API for Inference

• PyTorch Neuron (torch-neuronx) Weight Replacement API for Inference

• PyTorch NeuronX NeuronCore Placement APIs [Beta]

• PyTorch NeuronX Analyze API for Inference

• PyTorch NeuronX DataParallel API

• PyTorch NeuronX Lazy and Asynchronous Loading API

Developer Guide (torch-neuronx)

• NeuronCore Allocation and Model Placement for Inference (torch-neuronx)

• Comparison of Traced Inference versus XLA Lazy Tensor Inference (torch-neuronx)

• Data Parallel Inference on torch_neuronx

• Autobucketing for Inference (|torch-neuronx|)

Misc (torch-neuronx)

• PyTorch Neuron (torch-neuronx) release notes

This document is relevant for: Inf2, Trn1, Trn1n