This document is relevant for: Inf2, Trn1

Transformers NeuronX (transformers-neuronx)

Setup (transformers-neuronx)

If you already have setup your environment to run PyTorch NeuronX, you just need to install Transformers NeuronX library using the following instruction.

pip install transformers-neuronx --extra-index-url=https://pip.repos.neuron.amazonaws.com

If you are starting from scratch, Neuron Multi Framework DLAMI is recommended as it comes pre-installed with Transformers NeuronX virtual environment. You can refer to the instructions to launch a Neuron instance using Multi Framework DLAMI

Developer Guide (transformers-neuronx)

• Transformers NeuronX (transformers-neuronx) Developer Guide

Tutorials (transformers-neuronx)

• Hugging Face meta-llama/Llama-2-13b autoregressive sampling on Inf2 & Trn1

• Hugging Face facebook/opt-13b autoregressive sampling on Inf2 & Trn1

• Hugging Face facebook/opt-30b autoregressive sampling on Inf2 & Trn1

• Hugging Face facebook/opt-66b autoregressive sampling on Inf2

Misc (transformers-neuronx)

• Transformers Neuron (transformers-neuronx) release notes

This document is relevant for: Inf2, Trn1