Inference with torch-neuronx (Inf2 & Trn1/Trn1n)

Tutorials (torch-neuronx)

• HuggingFace pretrained BERT tutorial [html] [notebook]

• TorchServe tutorial [html]

• LibTorch C++ tutorial (for torch-neuron and torch-neuronx) [html]

• Torchvision ResNet50 tutorial [html] [notebook]

Note

To use Jupyter Notebook see:

• Jupyter Notebook QuickStart

• Running Jupyter Notebook as script

Additional Examples (torch-neuronx)

• AWS Neuron Samples GitHub Repository

• Transformers Neuron GitHub samples

API Reference Guide (torch-neuronx)

• PyTorch Neuron (torch-neuronx) Tracing API for Inference

• PyTorch Neuron (torch-neuronx) NeuronCore Placement APIs [Experimental]

• PyTorch Neuron (torch-neuronx) Analyze API for Inference

Developer Guide (torch-neuronx)

• NeuronCore Allocation and Model Placement for Inference (torch-neuronx)

• Comparison of Traced Inference versus XLA Lazy Tensor Inference (torch-neuronx)

Misc (torch-neuronx)

• PyTorch Neuron (torch-neuronx) release notes

Transformers Neuron (transformers-neuronx)

• Transformers Neuron (transformers-neuronx) Developer Guide